acceptodds
Under review as a conference paper at ICLR 2027

Policy-Adaptive Credit Assignment for Spatial Reasoning with Verifiable Rewards

Abstract

Image-based spatial reasoning requires perception, decomposition, and inference, but the stage that limits accuracy can change during training. Answer-level rewards neither localize failures to individual stages nor provide discriminative group-relative feedback when all rollouts receive the same reward. Process supervision provides intermediate feedback, but fixed weights cannot adapt reward allocation to these changing bottlenecks. To address this gap, we propose CALIPER, an adaptive process-reward framework supported by SpatialTrace, a corpus of stage-decomposed spatial reasoning traces. CALIPER jointly estimates the associations between stage-wise reference coverage and answer correctness through within-group ridge regression, using these associations as utility proxies to gradually update reward weights. The resulting process reward supervises all rollout groups, including uniformly failing ones. Because utility estimates can become unreliable when answer-discriminative groups are scarce, a closed-loop harness monitors estimator reliability and validation performance to guide checkpoint rollback, rollout diversification, and early stopping. Extensive experiments across eight benchmarks and three backbones show that CALIPER achieves the highest mean under both one-shot and harness training, with consistent gains over fixed allocation across RL optimizers. Further analyses confirm the benefits of adaptive reward allocation and harness control, with favorable scaling in model/data size.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial reasoning, vision-language models, verifiable-reward reinforcement learning, process supervision, adaptive reward allocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DD4wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

It has been proposed that visual reasoning accuracy improves if a judge LLM separately scores the accuracies of three stages of reasoning: perception, decomposition, and inference. This paper proposes that the weight given to each stage should be adapted based on the utility of that stage in discriminating between accurate versus inaccurate rollouts in recent groups.

Critical strengths
  • The use of utility-dependent weighting is shown to improve performance relative to uniform or fixed weighting
  • Proposition 1 shows that the proposed leaky integration of utility-dependent weighting (Eq. 4) guarantees a graceful degradation: the proposed allocation converges to uniform as the fraction of answer-discriminative groups goes to zero
Critical weaknesses
  • The proposed closed-loop harness decides between four actions (rollback, diversity, stop, and continue) that are widely used in most harnesses. Table 2 shows that, on some corpora, using this harness with PRM achieves results as good as the results of proposed CALIPER.
  • The proposed utility-dependent weighting has an impact on the timing of the harness' diversify action, but its impact is not separately measured using an ablation.
  • Figure 5 shows that many of the benefits of the proposed training algorithm come from steps that are pretty well-established and have been used by other authors, e.g., the independent judge of each segment, the physics prior, the solved-group gate.
  • Notation is often unnecessarily confusing. For example, why introduce a new letter (d) for |S| in Proposition 1, only to use that letter in exactly one location in Eq. (7)?
Questions for authors
  • Can you measure the impact of utility-dependent weighting on the harness?
2 comments
AuthorsOpenReview ↗

Authors' Rebuttal

Thank you for your feedback and valuable comments. Below is our point-by-point response.


W1 [Standard harness actions and the Harness PRM row of Tab. 2]

CALIPER's harness rules depend on its utility estimator, and the harness alone does not reproduce CALIPER's gains. We address this from two aspects below:

(1) Estimator-Dependent Rules. The paper presents the harness as an audit of the utility estimate rather than as a new control scheme. Eq. 5 conditions DIVERSIFY and STOP on the allocation drift δk\delta_k, the number nkMn_k^{\mathcal M} of answer-discriminative groups and the utility separation Δuk\Delta u_k, and ROLLBACK restores the policy, allocation and rollout temperature of the best validation round. With fixed weights, δk=0\delta_k=0 in every round, so the drift condition no longer indicates whether the allocation has stabilized. Prop. 1 shows that the pooled target converges to the uniform allocation as p↓0p\downarrow0, whereas CALIPER's mixed-group target does not depend on pp. CALIPER therefore keeps its stage contrast when answer-discriminative groups become scarce, and the update threshold Nmin⁡N_{\min} holds the current allocation until enough samples accumulate. nkMn_k^{\mathcal M} detects this scarcity and DIVERSIFY counteracts it. The same four actions lower PRM on every backbone (Tab. 1) and add at most 0.54 points under frozen allocation, but add 1.13 to 3.24 points under adaptive allocation (Table R1 in our response to W2), so their value depends on the estimator state on which they act.

(2) Gains beyond the Harness. Harness PRM uses "the separate harness protocol" (Sec. 5.1), whereas the CALIPER row of Tab. 2 is trained without it. Harness PRM matches CALIPER only on RealWorldQA, scores lower on BLINK, EmbSpatial and ViewSpatial, and its mean is 1.50 points lower. On the eight benchmarks of Tab. 1, harness training lowers PRM and raises CALIPER on all three backbones. Under the same harness, CALIPER exceeds PRM by 1.50, 3.77 and 2.06 points.


W2 & Q1 [Impact of utility-dependent weighting on the harness]

Utility-dependent weighting determines both the timing of the harness interventions and the gain they provide. We address this from two aspects below:

(1) Factorial Ablation. We train adaptive and frozen allocation each with and without the harness rules (Table R1).

Table R1. Eight-benchmark mean (%) under the harness protocol.

AllocationHarness rulesQwen3-VL-4BQwen2.5-VL-7BInternVL3-8B
Frozen✗62.2154.6860.73
Frozen✓62.4755.2260.96
Adaptive✗62.9353.0161.42
Adaptive (CALIPER)✓64.4056.2562.55

With at most 30 optimizer steps in both settings, the rules add 1.13 to 3.24 points under adaptive allocation but only 0.23 to 0.54 points under frozen allocation. The largest gain, 3.24 points on Qwen2.5-VL-7B, comes mostly from ROLLBACK, whose removal alone costs 2.17 points.

(2) Timing of DIVERSIFY. We repeat CALIPER and frozen allocation with three seeds and track the decision Dk\mathcal D_k of every round. Under frozen allocation, DIVERSIFY is triggered in 7 of 9 runs at round 2.3 on average. Under adaptive allocation, it is triggered in 5 of 9 runs at round 3.4 on average, after the allocation stabilizes. Under frozen allocation, the extra groups cannot change the fixed weights, consistent with the small rule gain in Table R1. We have added Table R1 and these timing statistics to App. D.


W3 [Contribution of established components in Fig. 5]

The frozen-allocation control isolates the gain from weight adaptation, and the component ablations identify measured utility as the largest single contributor on every backbone. The control preserves the judge and reference access (App. B.1), and CALIPER exceeds it by 1.93, 1.03 and 1.59 points (Fig. 5b), so components shared with the control cannot account for this margin.

In Fig. 5a, removing measured utility lowers the mean by 2.66, 2.45 and 1.73 points, whereas the largest drop from removing independent judging, the length penalty, the physics prior or the solved-group gate is 2.24, 1.71 and 1.16 points. Replacing measured utility with failure-only or dispersion-only heuristics also lowers the mean on every backbone, by up to 3.64 and 2.09 points. The gain therefore requires the regression estimate of stage utility rather than any adaptive weighting, and the supporting components are not among the paper's stated contributions.


W4 [Notation]

Prop. 1 and Eq. 7 now use ∣S∣|\mathcal S| instead of dd, which remains only in App. C, where the proof uses it throughout. The harness thresholds, which shared ϵ\epsilon with the allocation offset, now have distinct symbols, and App. B adds a notation table.


We hope this addresses your concerns and are happy to discuss further.

AuthorsOpenReview ↗

Follow-up on remaining concerns

As the discussion period draws to a close, we would appreciate knowing whether our responses have addressed your concerns. If so, we would be grateful if you could reconsider your rating accordingly. We would also be happy to address any remaining questions during the remaining discussion period.

Reviewer_un45OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1(−1)
Read review
Paper summary

The paper presents CALIPER, a method that dynamically adjusts rewards for perception, problem decomposition, and inference when training visual reasoning models. It learns which stages are most associated with correct answers and uses this information to guide training, even when all sampled answers are wrong. The authors also introduce SpatialTrace, a dataset with stage-by-stage reference reasoning. Experiments across several models and benchmarks show improvements over fixed reward weights, with ablations supporting the main design choices.

Critical strengths
  1. The experiment is very sufficient. CALIPER is evaluated on many representative benchmarks on spatial reasoning. The author also test different RL algorithm and conduct ablation on different reward components.

  2. The method is well-organized. Some incorrect answers still contain better intermediate reasoning. CALIPER can reward these differences when answer-only rewards provide no learning signal. The results show that better reward allocation matters more than simply adding process feedback.

Critical weaknesses
  1. The method depends heavily on trace quality. The traces are annotated by Qwen2.5-VL-32B and scored by DeepSeek-V4-Flash, so the ceiling is the trajectory that Qwen2.5-VL-32B produces after seeing the ground truth. The teacher itself is not strong enough. The human evaluation in Appendix A finds that fewer than 80% of the stage traces are acceptable, and the appendix example shows that when the model's perception conflicts with the ground truth, it fabricates a chain of thought to force agreement with the label ("Adding the height of the TV and stand (2.7 meters) to the estimated additional height of the fireplace (0.289 meters) gives a total height of 2.989 meters for the fireplace.")
  2. The method assumes that high-quality reasoning traces are already available. However, a main advantage of RLVR is that it needs no trace-level supervision and can train from verifiable answers alone. When such traces already exist, using RL complicates a problem that could be handled more directly. Why not use other methods such as SFT or on-policy distillation which needs trace? The paper does not include these methods as baselines.
Questions for authors
  1. Would generating the traces with a stronger closed-source model improve trace quality, and would that improve training performance?
  2. What happens if SpatialTrace is used directly to train the model with SFT or on-policy distillation?
4 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for your feedback and valuable comments. Below is our point-by-point response.


W1 & Q1 [Trace quality, the annotator ceiling and a stronger annotator]

SpatialTrace references are grading targets rather than imitation targets, and CALIPER weights each stage by its measured association with verified answers. Re-annotating with GPT-5 raises the mean positive stage rating from 76.9% to 86.8% but improves CALIPER's mean by at most 0.46 points, and corrupted references degrade CALIPER less than they degrade frozen allocation. We evaluate three reference variants. We re-annotate all items with GPT-5 under the same protocol, mask the stage references of flagged training items, and replace the inference reference of 25% or 50% of training items with that of another item from the same family. Table R1 reports the results.

Table R1. Eight-benchmark mean (%) under the harness protocol for different SpatialTrace references.

BackboneAllocationOriginalGPT-5Flagged masked25% replaced50% replaced
Qwen3-VL-4BFrozen α62.4762.9862.4161.5060.72
Qwen3-VL-4BCALIPER64.4064.8664.4763.9663.31
Qwen2.5-VL-7BFrozen α55.2255.6155.2654.5653.81
Qwen2.5-VL-7BCALIPER56.2556.6856.2755.7355.37
InternVL3-8BFrozen α60.9661.4360.9460.1559.13
InternVL3-8BCALIPER62.5562.9762.5062.2661.69

(1) Role of the References and Robustness to Corruption. The judge counts reference steps that have a semantically equivalent counterpart in the rollout (App. B.1), and this coverage term has weight wm=0.25w_m=0.25 shared by the three stages, compared with wa=0.60w_a=0.60 for the verified answer (Eq. 2, Tab. 7). Stage utility is the within-question association between coverage and answer correctness. With half of the inference references replaced, CALIPER loses at most 1.09 points, whereas frozen allocation loses at least 1.41 points, and CALIPER's final inference weight on Qwen3-VL-4B falls from 0.71 to 0.45. Under this corruption, CALIPER still exceeds frozen allocation trained with the original references. Reference errors therefore lower the weight of the affected stage instead of entering the reward at a fixed weight.

(2) Human Ratings and a Stronger Annotator. Tab. 6 counts only the highest rubric levels as positive: Faithful perception, Adequate decomposition and inference rated 4 or 5. A Partial perception that is correct but omits one object is therefore not positive. Unfaithful perception accounts for 5.6% of ratings, Inadequate decomposition for 4.3%, and inference rated 1 or 2 for 7.4%. The GPT-5 ratings above use the same 100 items and raters, and with these references CALIPER's margins over frozen allocation remain 1.88, 1.07 and 1.54 points.

(3) Omni3D Example and Flagged References. The record marks the answer alignment of this item as not supported, following the annotation protocol that records discrepancies between visual evidence and the source label (App. A). The value 2.989 m is the Omni3D-Bench label, which every method trained on this item, including outcome-only RLVR, is rewarded for reaching. The trace describes the 0.289 m increment as "a coarse visual estimate" for which "no exact measurement tools are available". Flagged items of this kind account for 11.6% of the training set, and masking their stage references changes CALIPER's mean by at most 0.07 points.

We have added the per-category ratings to App. A and Table R1 to App. D of the revised manuscript.


W2 & Q2 [Reliance on traces, and SFT or on-policy distillation on SpatialTrace]

CALIPER outperforms SFT, on-policy distillation and SFT followed by outcome-reward GRPO on all three backbones with the same SpatialTrace data, keeps the verifiable answer reward as its dominant term, and supports training with automatically generated traces, including incomplete ones. We address this from two aspects below:

(1) Comparison with SFT and On-Policy Distillation. We train each backbone on SpatialTrace with SFT, with SFT followed by outcome-reward GRPO, and with on-policy distillation from the annotator Qwen2.5-VL-32B conditioned on the reference trace. All variants use the same rank-64 LoRA adapter, training data and validation-based model selection as the harness protocol. Table R2 reports the results.

Table R2. Eight-benchmark mean (%) of untrained CoT and methods trained on SpatialTrace, with RL rows under the harness protocol.

MethodQwen3-VL-4BQwen2.5-VL-7BInternVL3-8B
CoT, untrained60.5154.2359.62
SFT60.1254.8759.95
On-policy distillation61.2455.8860.06
SFT, then outcome reward61.5855.9260.21
Outcome reward61.3655.6459.53
CALIPER64.4056.2562.55

Please see the subsequent Authors' Rebuttal (2/2) for remaining responses

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


CALIPER outperforms every alternative on all three backbones, by 0.33 to 4.28 points, and SFT falls below untrained CoT on Qwen3-VL-4B, a backbone released after the annotator. SFT maximizes the likelihood of every reference token, including unsupported estimates, and on-policy distillation moves the policy toward the teacher's distribution, so both are bounded by the annotator. CALIPER instead rewards coverage mainly to the extent that it predicts verified answers, so the answer verifier remains the final criterion. We have added the imitation rows of Table R2 to Tab. 3 and the comparison to Sec. 5.2 of the revised manuscript.

(2) Trace Requirements. SpatialTrace references are generated automatically by the open-source Qwen2.5-VL-32B from the image, question and answer, so CALIPER needs no human-written traces. Generating the references for all 1,180 items took 1.8 GPU-hours, below 4% of the cost of one harness training run. Of the 1,000 training items, 156 lack a complete trace and receive answer reward and process reward on their available stages. For an item without references, the process term of Eq. 2 vanishes and the item is supervised through its answer reward as in outcome-only RLVR, so CALIPER adds a grading signal to answer-level training rather than replacing it. Under one-shot training with complete traces for only 25% of items, CALIPER reaches 61.10 on InternVL3-8B (Fig. 8d), above outcome reward (58.82) and comparable to PRM (61.15), whose stage labels derive from the full set of references (Tab. 1).


We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

AuthorsOpenReview ↗

Follow-up on remaining concerns

As the discussion period draws to a close, we would appreciate knowing whether our responses have addressed your concerns. If so, we would be grateful if you could reconsider your rating accordingly. We would also be happy to address any remaining questions during the remaining discussion period.

Thank you for the response, but your reply actually deepens my concerns rather than resolving them. I would appreciate your clarification on the following three questions:

  1. Are the reference traces actually providing any real supervision? Looking at Table R1 and Fig. 8d, the model seems almost completely insensitive to trace quality. When you re-annotate with GPT-5 and boost human approval by ~10 points (76.9% to 86.8%), the final score improves by less than 0.5 points (even Frozen α\alpha improves by 0.4–0.5). Besides, masking out the 11.6% low-quality traces (mentioned in W1(3)) only changes the score by 0.07 points. Even more surprisingly, the Inference stage accounts for about 60% of your process reward weight, yet replacing 50% of the Inference references with completely unrelated traces from other questions only drops Qwen2.5-VL-7B and InternVL3-8B by ~0.87 points. And in Fig. 8d, dropping 75% of the complete traces only costs 0.74 points. If making the traces 10% better barely helps, and replacing half of them with wrong traces from other questions barely hurts, what role are the reference traces actually playing? Right now it looks like the process reward is not teaching step-level spatial reasoning at all. If the semantic content of the traces matters this little, what is the point of SpatialTrace and making 4,096 online DeepSeek-V4-Flash judge calls per step?

  2. The SFT comparison in Table R2 is unfair—why not train SFT on the GPT-5 traces? On Qwen2.5-VL-7B, simple SFT, then outcome reward already reaches 55.92%, nearly equal to CALIPER (56.25%), yet doesn't need online LLM judge, ridge regression, or multi-round rollbacks. More importantly, you trained the SFT baselines on the raw Qwen2.5-VL-32B dataset, which contains 15.6% incomplete traces and 11.6% low-quality/fabricated items. Since you already re-annotated the dataset with GPT-5 for Table R1, why didn't you run SFT and SFT, then outcome reward on the cleaner GPT-5 traces? Also, since the original gpt-5 (gpt-5-2025-08-07) has been deprecated by OpenAI since June, which exact API model snapshot did you use here?

  3. How did you actually implement On-Policy Distillation (OPD)? In W2(1), you state that you ran "on-policy distillation from the annotator Qwen2.5-VL-32B conditioned on the reference trace" across all three backbones. How was this implemented in practice?

  • First, although InternVL3-8B and Qwen3-VL-4B share the base Qwen2 vocabulary with Qwen2.5-VL-32B, their vocabulary sizes / LM head dimensions are different (152,064 for Qwen2.5-VL-32B, 151,936 for Qwen3-VL-4B, and 151,674 for InternVL3-8B). How did you handle the vocabulary size mismatch?
  • Second, Appendix A states that Qwen2.5-VL-32B is served via Ollama. As far as I know, Ollama cannot return full-vocabulary logits or prefill distributions over student rollouts. How did you obtain the teacher logits for OPD?

I look forward to your concrete clarification on these points; if these concerns cannot be convincingly resolved, I will have to lower my score.

Reviewer_AEPAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how process rewards can be allocated across perception, decomposition, and inference during reinforcement learning for visual spatial reasoning. The authors construct SpatialTrace by adding stage-specific reasoning references to questions sampled from existing datasets. The proposed framework, CALIPER, uses within-group ridge regression to estimate the association between stage coverage and answer correctness from rollout groups containing both correct and incorrect answers. These estimates guide updates to the stage weights used to reward all rollout groups, including uniformly incorrect ones. An outer control loop monitors validation performance and estimated diagnostic results, thereby guiding subsequent training.

The paper evaluates CALIPER on eight benchmarks with three vision-language backbones . The authors report improvements over fixed reward allocation across GRPO, DAPO, and GSPO, although process-based training does not consistently outperform outcome-only rewards. Additional experiments examine generalization to other benchmarks, reward allocation dynamics, component ablations, hyperparameter sensitivity, and scaling with model size, training data, update budget, and reference coverage.

Critical strengths

S1. The distinction between stage quality and its association with answer correctness is useful. Section 4 estimates this association through within-question comparisons rather than directly treating low coverage as evidence that a stage deserves more reward. Proposition 1 provides a concrete rationale for estimating utility from mixed-correctness groups, while applying the resulting rewards to all groups. This gives the allocation mechanism a clearer basis than weighting stages solely by their scores or failure rates.

S2. The evaluation includes controls that help isolate the contribution of adaptive allocation. The frozen-allocation comparison preserves reference access and the reward judge, making it more informative than comparisons against outcome-only rewards or a separately trained PRM. These results provide meaningful evidence that adapting the weights contributes beyond simply adding process supervision.

Critical weaknesses

W1. My main concern is that the empirical improvements are modest and their reliability is not sufficiently established. Table 3 shows that it does not consistently outperform simpler outcome-only training across optimizers. It remains unclear under which training conditions the proposed allocation provides a meaningful advantage. Moreover, the main results lack repeated-run variability or confidence intervals, making it difficult to assess whether the relatively small gains are consistent across training runs and evaluation samples.

W2. The role of the closed-loop harness is not clearly analyzed. The experiments mainly report final performance with and without the combined rules, without showing when individual interventions are triggered or how they affect subsequent training. It remains unclear which harness rules contribute to the reported gains and whether increasing sampling diversity can produce more informative rollout groups.

Questions for authors

Q1. What motivates the three-stage division into perception, decomposition, and inference? Is it based on prior work or empirical observations, and what benefit comes from treating decomposition and inference as separate stages?

Q2. Could the authors report results across multiple training seeds, together with uncertainty estimates for the main comparisons?

Q3. Could the authors separately examine the effects of rollback, rollout diversification, and stopping, and report when these rules are triggered? In particular, does increasing sampling diversity produce more informative rollout groups when they become scarce?

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for providing us with your valuable feedback. We have carefully considered your comments and would like to address them as below.


W1 & Q2 [Reliability of the gains and the comparison with outcome-only training]

CALIPER improves on frozen allocation in all nine optimizer and backbone settings of Tab. 3 and in every GRPO seed. Its margin over outcome-only training is largest under GRPO and on items that the untrained backbone answers incorrectly in all eight samples, and the two settings in which outcome-only training achieves a higher mean are both on Qwen2.5-VL-7B. We address this from three aspects below:

(1) Comparison with Frozen Allocation and Outcome Reward. The frozen-allocation control preserves the judge and reference access, "isolating the contribution of weight adaptation" (App. B.1). Under harness training, CALIPER exceeds it by 1.93, 1.03 and 1.59 points (Tab. 3), which is more than process supervision itself adds over outcome reward at fixed weights. The margin is 1.99, 1.55 and 2.85 points on the five spatial benchmarks (Tab. 9) and 3.00 points on the Qwen3-VL-4B OOD benchmarks under one-shot training (Tab. 2). CALIPER exceeds outcome-only training in 7 of 9 settings. The two exceptions are DAPO and GSPO on Qwen2.5-VL-7B, where reference-assisted reward and frozen allocation are below outcome reward under all three optimizers, and only CALIPER, under GRPO, exceeds outcome reward. Under DAPO and GSPO, CALIPER narrows the gap between frozen allocation and outcome reward by 0.60 and 0.92 points, respectively. As Sec. 5.2 states, adaptive allocation improves process supervision without requiring process rewards to dominate outcome rewards.

(2) Variability across Training Runs. We retrain the four main methods with two additional seeds. Table R1 reports the results.

Table R1. Eight-benchmark mean (%) ± standard deviation over three seeds under the harness protocol, and CALIPER's differences with 95% bootstrap confidence intervals over seeds and evaluation items.

BackboneOutcomePRMFrozen αCALIPERΔ vs. Frozen αΔ vs. Outcome
Qwen3-VL-4B61.40 ± 0.2962.81 ± 0.5262.45 ± 0.1864.30 ± 0.44+1.85 [0.74, 3.05]+2.90 [1.76, 4.02]
Qwen2.5-VL-7B55.61 ± 0.6152.54 ± 0.2455.16 ± 0.4756.17 ± 0.31+1.01 [0.06, 2.12]+0.56 [−0.62, 1.63]
InternVL3-8B59.53 ± 0.2260.48 ± 0.3960.92 ± 0.5662.52 ± 0.27+1.60 [0.43, 2.66]+2.99 [1.78, 4.27]

Seed standard deviations are smaller than every margin over frozen allocation, CALIPER exceeds frozen allocation in every seed on every backbone, and all three intervals for this margin exclude zero. Against outcome reward, the intervals exclude zero on Qwen3-VL-4B and InternVL3-8B.

(3) Conditions for the Margin over Outcome Reward. Process reward provides a learning signal mainly in outcome-degenerate groups, whose outcome advantages vanish (Eq. 1). Consistent with this mechanism, the margin is largest under GRPO on every backbone, whereas DAPO, which filters outcome-degenerate groups online, raises the outcome baseline by 1.95, 0.67 and 1.81 points over GRPO. To relate the margin to item difficulty, we group evaluation items by how often the untrained backbone answers them correctly. Table R2 reports the differences in each group.

Table R2. CALIPER's differences (points) on evaluation items grouped by the number of correct answers among eight samples from the untrained backbone, pooled over all backbones and seeds.

Correct samplesShare of itemsΔ vs. OutcomeΔ vs. Frozen α
0 of 822%+6.4+3.7
1 to 7 of 838%+2.3+1.6
8 of 840%−0.3+0.2

The gains concentrate on items with no correct answer among the eight samples. We have added Tables R1 and R2 to App. D of the revised manuscript.


W2 & Q3 [Individual harness rules, when they are triggered, and the effect of diversification]

ROLLBACK and DIVERSIFY account for most of the gain from the harness rules. DIVERSIFY restores answer-discriminative groups when they become scarce, and STOP reduces the number of optimizer steps by about 18% while changing the mean by at most 0.21 points. We address this from two aspects below:

(1) Contribution and Timing of Each Rule. We remove one rule at a time with the same seed as Tab. 9. Table R3 reports the results.

Table R3. Eight-benchmark mean (%) under the harness protocol with one rule removed, and how often that rule is triggered in the nine full CALIPER runs of Table R1.

VariantQwen3-VL-4BQwen2.5-VL-7BInternVL3-8BTriggered in full runs
All rules (CALIPER)64.4056.2562.55
w/o ROLLBACK63.7154.0862.125 of 9
w/o DIVERSIFY63.5655.6261.835 of 9
w/o STOP64.2256.0462.473 of 9
No rules62.9353.0161.42

Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


(2) Effect of Diversification on Rollout Groups. Each time DIVERSIFY is triggered, we also continue training from the same round without the temperature increase and compare the following round. Table R4 reports the results.

Table R4. Statistics of the round following DIVERSIFY, averaged over the five runs in which it is triggered.

Statistic in the following roundWithout increaseWith increase
Answer-discriminative groups per batch, nkMn^{\mathcal M}_k0.61.9
Steps meeting the update threshold Nmin⁡N_{\min} (%)4188
Change in validation accuracy (points)+0.2+1.1

Raising the temperature increases nkMn^{\mathcal M}_k from 0.6 to 1.9, above the threshold νmin⁡=1\nu_{\min}=1 of Eq. 5, and most allocation updates again meet the update threshold. Starting training at temperature 1.5 lowers the mean by 0.91 and 1.27 points on Qwen3-VL-4B and Qwen2.5-VL-7B (Tab. 9), so raising the temperature helps when answer-discriminative groups become scarce, rather than throughout training. We have added the single-rule ablations to Fig. 5b and Tables R3 and R4 to App. D of the revised manuscript.


Q1 [Motivation for the three-stage division]

The division follows established decompositions of visual reasoning, and separating decomposition from inference allows CALIPER to assign them different reward weights and outperforms a merged reasoning stage. Visual programming methods separate perception modules from the composition of subproblems and their execution, as in VisProg [1] and ViperGPT [2]. Least-to-Most prompting [3] likewise treats reducing a question to subquestions and solving them as distinct stages. SpatialTrace follows this structure with perception, decomposition and inference.

We also merge decomposition and inference into a single reasoning stage with all other settings fixed. Table R5 reports the results.

Table R5. Eight-benchmark mean (%) of CALIPER under the harness protocol with two or three stages.

Stage divisionQwen3-VL-4BQwen2.5-VL-7BInternVL3-8B
Two stages (perception, merged reasoning)63.3855.5461.67
Three stages (CALIPER)64.4056.2562.55

Merging lowers the mean by 1.02, 0.71 and 0.88 points. The two stages also receive different weights: starting from uniform weights, the inference weight rises to between 0.58 and 0.69 whereas the decomposition weight remains between 0.17 and 0.28 (Fig. 4), and the within-group correlation between their coverage scores is only 0.41, so a shared weight would not reflect this difference. Fig. 7 shows the two failure types separately. With the same objects perceived, the baselines fail to infer the Omni3D item and to decompose the RoboSpatial item. We have added these citations to Sec. 3 and Table R5 to App. D of the revised manuscript.

Reference:

[1] Visual programming: Compositional visual reasoning without training. CVPR 2023

[2] Vipergpt: Visual inference via python execution for reasoning. ICCV 2023

[3] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR 2023


We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

AuthorsOpenReview ↗

Follow-up on remaining concerns

As the discussion period draws to a close, we would appreciate knowing whether our responses have addressed your concerns. If so, we would be grateful if you could reconsider your rating accordingly. We would also be happy to address any remaining questions during the remaining discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers, AC, SAC, and PC,

Thank you for your time and effort in reviewing and handling our paper. We have posted point-by-point responses to each reviewer and summarize the main points below.


Paper Overview and Strengths

[Paper Overview]: This paper proposes CALIPER, an adaptive process-reward method for RLVR on image-based spatial reasoning. Answer-level rewards cannot localize failures to perception, decomposition or inference, and give no signal when all rollouts in a group receive the same reward. CALIPER scores each stage against SpatialTrace reference traces and weights it by its measured association with answer correctness, audited by a closed-loop harness. It attains the highest average performance across eight benchmarks and three backbones and improves on fixed allocation under GRPO, DAPO and GSPO.

[Strengths]: We are encouraged that the reviewers recognize the distinction between stage quality and its association with answer correctness (AEPA), the theoretical support of Prop. 1 (AEPA, DD4w), the frozen-allocation control (AEPA), the extensive evaluation and well-organized method (un45), and the evidence that adapting the weights contributes beyond adding process supervision (AEPA, un45) and improves on uniform or fixed weighting (DD4w).


Additional Experiments for Concerns

  • [Seeds and Confidence Intervals] (AEPA): We retrain the four main methods with three seeds. CALIPER exceeds frozen allocation in every seed on every backbone, and the bootstrap intervals exclude zero (Table R1 for AEPA).
  • [Conditions for the Gain] (AEPA): We group evaluation items by initial difficulty. The gains concentrate on items that the untrained backbone answers incorrectly in all eight samples, +6.4 points over outcome reward and +3.7 over frozen allocation (Table R2 for AEPA).
  • [Harness Rules] (AEPA, DD4w): We ablate each rule, report when it is triggered, and compare the round after DIVERSIFY with and without the temperature increase (Tables R3 and R4 for AEPA). The rules add 1.13 to 3.24 points under adaptive allocation but at most 0.54 under frozen allocation (Table R1 for DD4w).
  • [Two versus Three Stages] (AEPA): Merging decomposition and inference into one stage lowers the mean by 0.71 to 1.02 points (Table R5 for AEPA).
  • [Reference Quality] (un45): GPT-5 re-annotation raises the positive stage rating from 76.9% to 86.8% but improves CALIPER by at most 0.46 points, masking flagged references changes it by at most 0.07 points, and replacing half of the inference references degrades CALIPER less than frozen allocation (Table R1 for un45).
  • [SFT and On-Policy Distillation] (un45): CALIPER outperforms SFT, on-policy distillation and SFT followed by outcome-reward GRPO trained on the same data, by 0.33 to 4.28 points on every backbone (Table R2 for un45).

Summary of Responses to Reviewers' Questions

  • [Comparison with Outcome Reward] (AEPA): CALIPER improves on frozen allocation in all nine optimizer and backbone settings and on outcome reward in seven. Both exceptions are on Qwen2.5-VL-7B, where reference-assisted reward and frozen allocation are below outcome reward under all three optimizers (Sec. 5.2).
  • [Role of the References] (un45): References are grading targets rather than imitation targets, with weight 0.25 compared with 0.60 for the verified answer. Items without references are trained from the answer reward as in outcome-only RLVR, and generating all references took 1.8 GPU-hours.
  • [Harness Actions and Harness PRM] (DD4w): DIVERSIFY and STOP depend on estimator statistics, ROLLBACK restores the allocation with the policy, and Harness PRM remains 1.50 points below one-shot CALIPER in Tab. 2 and 1.50 to 3.77 points below CALIPER under the same harness.
  • [Established Components] (DD4w): Components shared with the frozen-allocation control cannot explain CALIPER's margin of 1.93, 1.03 and 1.59 points over it, and removing measured utility causes the largest drop among the ablated components.
  • [Prop. 1] (DD4w): The pooled target converges to the uniform allocation as answer-discriminative groups vanish, whereas CALIPER's mixed-group target is independent of their share.

Revisions in the Updated Manuscript

All changes are marked in blue. Sec. 3 adds the motivation and citations for the three-stage division, and Fig. 5b the single-rule ablations (AEPA). Sec. 5.2 and Tab. 3 add the comparison with SFT and distillation (un45). Prop. 1 and Eq. 7 use ∣S∣|\mathcal S| in place of dd, the harness thresholds receive distinct symbols, and App. B adds a notation table (DD4w). App. A reports the per-category human ratings (un45). App. D contains Tables R1 to R5 for AEPA, Table R1 for un45, and Table R1 for DD4w.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.