acceptodds
Under review as a conference paper at ICLR 2027

Shared-Reference Anchoring and Reference Separation for Chart Reasoning Verification

Abstract

Reliable multimodal reasoning verification requires judgments to remain grounded in the source image even when intermediate references are imperfect. Existing verifiers increasingly use model-generated references to make visual premises explicit, but reusing the same fallible reference across multiple verification paths can couple their errors and redirect otherwise distinct judgments toward the same false premise. We identify this failure as shared-reference anchoring and examine its causal effect by varying only the reference while holding the chart, question, and candidate trajectories fixed. To mitigate this dependency, we propose RefSplit, a reference-separated verification method that retains reference-conditioned reasoning while withholding the shared reference from a complementary chart-grounding path using the same verifier checkpoint. On 300 Value Position pairs and 300 Legend Binding pairs, RefSplit reduces the preference-accuracy drop under aligned false references from 26.67 to 4.17 points and from 7.50 to 0.33 points, respectively. The same failure also arises under ordinary generation, where 26.46% of 2,509 audited incorrect-reference value-error pools contain an incorrect candidate aligned with the false reference, and RefSplit reduces aligned-error selection by 8.43 points. Across five public chart benchmarks and five candidate policies, RefSplit achieves 53.02% mean Best-of-8 accuracy, the highest among the evaluated prior selectors and within 0.03 points of the matched non-separated control. These results demonstrate that access to the source image alone does not eliminate the influence of a shared fallible reference. By separating reference access across complementary verification paths, RefSplit limits shared error propagation while preserving candidate-selection quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Reasoning Verification, Vision-Language Models, Chart Reasoning, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7yjwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Shared-Reference Anchoring in multimodal chart reasoning verification, where multiple verification paths rely on the same erroneous reference, leading to correlated incorrect judgments. The authors propose RefSplit, which mitigates error propagation by allowing one verification path to use structured reference information while the other directly verifies reasoning against the source chart without access to the shared reference.

Critical strengths
  • The paper investigates the correlated judgment biases caused by multiple verification paths sharing the same erroneous reference, addressing a meaningful problem in improving the reliability of multimodal reasoning verification.
  • The authors find that incorrect textual references can significantly influence verification judgments even when the verifier still has access to the original chart, which is an interesting observation.
  • Furthermore, RefSplit effectively mitigates the influence of erroneous references without introducing additional verifier models or increasing the number of verification calls, while maintaining comparable performance to the non-separated baseline on standard benchmarks.
Critical weaknesses
  • The core design of RefSplit simply removes the shared reference from one verification path, allowing it to evaluate reasoning directly against the original chart. Although the authors demonstrate the effectiveness of this strategy in specific error scenarios, further evidence is needed to establish the additional benefits of RefSplit over direct visual verification alone, as well as the necessity of its dual-path design.
  • Moreover, the observation that multiple verification paths can exhibit correlated biases when relying on the same erroneous reference, and that removing this reference can mitigate such biases, is relatively intuitive. Therefore, the novelty and technical depth of the proposed approach appear limited.
Questions for authors

Same as weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7yjw, Overview and Point 1/4

We sincerely thank the reviewer for the thoughtful summary and for noting that RefSplit adds no verifier model and no verification calls over the non-separated baseline. The review helped us make the motivation and evidence for our design much more explicit. For each comment raised by the reviewer, we present new analyses and experiments carried out during the discussion phase, all of which will be added to the main text and appendix by the end of this phase.

We address the reviewer's comments across four points (Point 1 below, Point 2 in the second comment, and Points 3–4 in the third). If we have overlooked anything, we would be grateful if the reviewer could let us know.

  • Point 1. Additional benefit of RefSplit over direct visual verification alone
  • Point 2. Necessity of the dual-path design
  • Point 3. Whether the finding is intuitive, and novelty and technical depth
  • Point 4. Making the paper easier to follow

1. Additional benefit of RefSplit over direct visual verification alone

We agree that VV is a strong baseline, and the paper does not claim that RefSplit beats VV on the unstratified public average, where the +0.09 pp difference is not statistically significant. Rather, the reference path adds accuracy when the extracted reference is correct. Because Section 4.3 showed this using component accuracies on the controlled set, we evaluated the full selector directly on both the controlled set and candidate pools from ordinary generation, as well as against compute-matched reference-free controls.

New Evidence That the Reference Path Adds InformationResult
RefSplit − VV, controlled set, 711 questions with a correct reference+1.91 pp, significant
RefSplit − VV, 5,265 natural pools from ordinary generation with a correct reference+0.67 pp, significant
R−VR - V, same 5,265 natural pools+1.11 pp, significant
RefSplit − VV with the value field of the reference corrected (an oracle ceiling), 2,509 natural wrong-reference pools+5.34 pp, significant
RefSplit − (VV + 2nd reference-free view at comparable compute), all 938 controlled questions (paraphrased VV or 1.5× resolution VV)+1.46 pp and +1.53 pp, both significant
The same two reference-free controls on the five-benchmark public mean+0.30 pp, significant, and +0.08 pp, not significant
Within-question rank correlation with VV for RR, a paraphrased VV, and a 1.5× resolution VV0.59, 0.94, and 0.88

The gain over VV is therefore present in ordinary generation as well as on the controlled set, and it cannot be reproduced by spending the same compute budget on additional reference-free verification. At the same time, the trade-off is clear. When the reference is wrong, VV is more accurate (Section 4.3), and RefSplit requires 17 calls compared with 8 for VV (5.52 s per question against 2.48 s, or 2.22×). RefSplit is therefore designed for pipelines that use an extracted reference. In Point 2, we show why such pipelines require two separated paths. These analyses will be added to the appendix.

AuthorsOpenReview ↗

Response to Reviewer 7yjw, Point 2/4

2. Necessity of the dual-path design

Two paths are needed because a single path is either steered by reference errors or blind to structured reference information, and separated access is needed because two paths that share an erroneous reference fail together. In Section 4.2, we showed that an aligned false reference lowers preference accuracy by 26.67 pp when both paths receive it and by 4.17 pp under separated access at the exact same 17 calls. To test both requirements directly, we ran three new analyses (and will add a concise summary of this design to Section 3).

First, we compared one-path and two-path selectors with and without the reference on our submitted checkpoint, including a new separated variant VE+VV_E + V in which VEV_E reads the reference in place of RR while VV remains reference-free.

SelectorPaths Reading the ReferencePublic MeanControlled Set, AllSteering, Value Position
VVnone52.93%87.29%0.00 pp
VEV_E aloneits only path53.28%88.43%−24.50 pp
R+VER + V_Eboth53.05%87.36%−26.67 pp
RefSplit, R+VR + Vone of two53.02%87.28%−4.17 pp
VE+VV_E + V, new variantone of two53.09%88.99%−1.67 pp

Each simpler alternative falls short on one of these two goals. Using VEV_E alone works well on clean references but loses 24.50 pp under an aligned false reference. Using VV alone avoids steering but loses the benefit of correct references (−1.91 pp compared with RefSplit and −2.92 pp compared with VE+VV_E + V). Sharing the reference across two paths in R+VER + V_E still suffers a 26.67 pp drop. Only the two-path designs with separated access retain the reference while staying robust. Notably, the VE+VV_E + V variant matches RefSplit on the public mean, is 1.01 pp more accurate on correct references in the controlled set, and exceeds VV across the entire controlled set by 1.71 pp. We will report this variant in the appendix as additional evidence that the benefit extends beyond the specific choice of the reference-reading path and follows from the separation principle itself.

Second, we measured how sharing a reference couples failures across the two paths. Under the aligned false reference, RR fails on 98 of the 300 Value Position pairs. VEV_E, which reads the same false reference, fails on 79 of those 98 pairs, whereas VV, which does not read the reference, fails on only 2 of the 98. A second verification judgment can correct the first only if it does not fail on the same examples.

Third, we verified that this dual-path separation benefit holds across fusion weights, checkpoints, model families, and naturally occurring errors.

Robustness CheckResult
Fusion weight λ\lambda from 0.25 to 4Non-separated steering stays between −25.00 and −27.83 pp, while separated steering falls to −0.50 pp
Seven checkpoints of our backboneSeparation effect is positive in all seven, and +17.67 to +34.00 pp on Value Position across five new fine-tuning runs
Three other model families without our training (Qwen2.5-VL-7B, Gemma-4-12B, InternVL3.5-8B)Separation effect is +17.83 to +27.50 pp on Value Position and +26.67 to +31.17 pp on Legend Binding
Naturally occurring aligned errors, 664 poolsSelection of the aligned wrong candidate falls from 77.87% to 69.44%, and accuracy rises from 4.82% to 6.53%
All 2,509 incorrect-reference value-error pools from ordinary generationSeparation raises accuracy by 0.83 pp

Separating the two paths therefore lets the verifier use an explicit reference while keeping one grounding judgment free from reference errors.

AuthorsOpenReview ↗

Response to Reviewer 7yjw, Points 3–4/4

3. Whether the finding is intuitive, and novelty and technical depth

We agree that, in hindsight, the advice not to share a faulty input sounds natural. However, intuition alone does not explain why the chart-grounded path VEV_E, which sees the source chart directly and is instructed to check it, still follows a false text reference. Prompted by the reviewer's comment, we tested four simpler explanations for this behavior, and none of them accounts for the drop.

First, a path that sees the chart still follows the false text across four model families.

Path, Value Position, 300 Pairs, Table 2Unrelated ReferenceAligned False Reference
RR, instructed to treat the reference as given95.33%66.67%
VEV_E, instructed only to check the chart, also receives the reference96.50%72.00%
VV, never receives the reference97.17%97.17%

VEV_E loses 24.50 pp even though the chart is in front of it and its prompt is the exact same chart-grounding instruction used by VV. In new experiments, the same collapse occurs without any of our training on three other model families, with VEV_E steering of −43.67 pp on Qwen2.5-VL-7B, −35.33 pp on Gemma-4-12B, and −63.17 pp on InternVL3.5-8B. Furthermore, adding the reference line to the released external verifier VisualPRM causes it to lose 29.83 pp in the same comparison (falling from 86.33% to 56.50%).

Second, the drop does not happen because the chart is unreadable, as the verifier recognizes that the false record contradicts the chart when asked directly. In a new experiment, we asked the same checkpoint whether standalone records are supported by the chart without any candidate trajectory. Across 400 false records (which include all 300 aligned false records from the intervention) and 400 true records on Legend Binding, the checkpoint assigns a mean grounded probability of 0.012 to false records and 0.963 to true ones. Thus, the verifier easily detects that the record is false in isolation, but when the same record is supplied as context for judging a trajectory, it still lowers VEV_E's preference for the correct trajectory.

Third, explicitly instructing the verifier to check the reference against the chart first does not fix the problem. We added a sentence to the VEV_E prompt warning that the record was extracted automatically and may contain errors and asking the verifier to check each field against the chart before using it. Under this instruction, Value Position steering remains at −26.83 pp, compared with −24.50 pp under the standard prompt.

Fourth, the accuracy drop is driven by agreement with a wrong candidate, not by the plausibility of the reference error. Across 290 Value Position pairs, replacing the aligned false value with another value plotted elsewhere in the same chart (mentioned by neither candidate) leaves VEV_E accuracy at 94.66%, whereas the candidate-aligned false value drops it to 71.38%. An empty reference line has no detectable effect either (92.33% compared with 92.50%). A reference error is therefore harmful specifically when a candidate makes the same error, which occurs naturally in 26.46% of the 2,509 incorrect-reference value-error pools we audited under ordinary generation.

None of these four outcomes follows from simple intuition. A verifier that sees the chart, recognizes the false record in isolation, and is explicitly told to check the chart still follows the record when a candidate agrees with it. Our technical contribution lies in isolating this directional dependency and removing it through a matched design that holds the checkpoint, prompt interface, candidates, and 17-call budget fixed. These analyses will be added to the appendix.

4. Making the paper easier to follow

We agree that readers currently have to track the three verification paths and their inputs across sections without a single reference table, so we will make four concrete revisions to improve readability:

  • A compact notation table at the start of Section 2 will list, for RR, VEV_E, and VV, the inputs each path receives, its instruction, and its role, alongside the four reference conditions.
  • Section 3 will open with a three-sentence summary of the design before introducing the equations.
  • Appendix C will report VV and R+VER+V_E alongside RefSplit with paired bootstrap tests in a unified table so that all selector comparisons can be read in one place.
  • Each main-text section will include explicit pointers to the supporting analyses in the appendix.

We would be grateful for pointers to any passage that remains hard to follow.

Reviewer_7yjwOpenReview ↗

Response to Author

Hi, apologies for my late response.

Thank you for your response and the additional experiments. While the new results address some of my concerns, I still have reservations about the novelty of RefSplit, as its core design appears relatively straightforward. I am not yet fully convinced that the proposed approach offers sufficient methodological novelty beyond separating reference access between two verification paths.

Reviewer_r5LKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates shared-reference anchoring in chart reasoning verification and proposes RefSplit, which combines reference-conditioned reasoning verification with reference-free chart verification. Through controlled interventions and experiments on five chart benchmarks, the authors demonstrate that separating reference access reduces sensitivity to incorrect references while maintaining competitive candidate-selection performance.

Critical strengths

Meaningful problem identification. The paper identifies shared-reference anchoring as a relevant failure mode in multimodal reasoning verification, highlighting how incorrect intermediate references can introduce correlated verification errors. Convincing controlled experiments. The matched reference interventions effectively isolate the influence of incorrect references. The analysis of naturally occurring errors further supports the practical relevance of the identified problem.

Critical weaknesses

Limited methodological novelty. RefSplit primarily separates reference access between two verification paths and combines their scores. The technical contribution beyond existing grounded verification and multi-view verification approaches is not sufficiently established.

Marginal practical improvement. RefSplit achieves 53.02% average accuracy, only 0.09 percentage points above reference-free verification and slightly below the non-separated variant (53.05%). Although the method improves robustness against aligned false references, its advantage over the simpler reference-free baseline remains unclear. This limits the practical significance of the proposed framework.

Questions for authors

Given the marginal improvement over reference-free verification, what is the main practical benefit of retaining the additional reference-conditioned path? Can the authors provide statistical significance and computational cost comparisons against the reference-free baseline? How sensitive is RefSplit to the fusion weight? Since the reliability of each path varies with reference correctness, could a fixed weight undermine the benefits of reference separation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer r5LK, Overview and Point 1/5

We sincerely thank the reviewer for recognizing our problem identification and controlled experiments as strengths, and for the constructive critique, which helped us strengthen our argument. Below, we present new analyses and experiments from the discussion phase, all of which will be added to the paper by the end of this phase.

We address the reviewer's questions across five points (Point 1 below, Points 2–3 in the second comment, and Points 4–5 in the third). If we have overlooked anything, please let us know.

  • Point 1. Technical contribution beyond grounded and multi-view verification
  • Point 2. Practical benefit of retaining the reference-conditioned path
  • Point 3. Statistical significance and computational cost relative to the reference-free baseline
  • Point 4. Sensitivity to the fusion weight
  • Point 5. Whether a fixed weight undermines the benefits of separation

1. Technical contribution beyond grounded and multi-view verification

We agree that RefSplit is mechanically simple. Our main contribution is isolating and removing a specific dependency between verification paths. In Section 2.2, an aligned false reference steers the chart-grounded path VEV_E from 96.50% down to 72.00% even though the source chart is visible. In Section 4.2, separating reference access cuts the fused drop from 26.67 pp to 4.17 pp using the exact same 17 calls. The simplicity of the remedy is intentional, because adding model capacity, training data, or verification calls would confound the attribution of selection errors to shared reference access. To clarify how this differs from prior verifiers and rule out simpler causes, we added a prior-work comparison and two new experiments.

First, while prior work reuses extracted evidence across steps or candidates (such as EVPV and Perception-R1) and examines noisy references or priors (Perceval, TIM-PRM, EVPV, SGV, BIRCH, and Perception-R1, whereas VisualPRM uses no reference), their robustness checks inject reference errors at random via flips, shuffles, drops, or noise. None of the papers we examined constructs a false reference aligned with a specific wrong candidate while holding the chart, question, and candidates fixed, or contrasts two verification paths that differ solely in reference access under the same checkpoint and call budget.

Second, we tested whether any plausible false value from the chart is enough to steer VEV_E. For 290 of the 300 Value Position pairs, we replaced the aligned false value with another value plotted elsewhere in the same chart that neither candidate mentions and whose distance from the true value is closest to that of the aligned value.

Path, 290 PairsUnrelated ReferencePlotted Non-Aligned False ValueAligned False Value
VEV_E96.38%94.66%71.38%
RR95.17%92.93%65.86%
RefSplit96.55%97.59%92.59%

A plotted false value that no candidate uses has little effect on VEV_E (−1.72 pp, not significant), whereas the candidate-aligned false value reduces accuracy by an additional 23.28 pp. A reference error is therefore harmful specifically when a candidate repeats the same error.

Third, we tested whether standard reference-free multi-view verification has the same effect as separated reference access. We paired VV with a paraphrased-instruction view VPV_P and with a 1.5× higher-resolution view VZV_Z, using rank correlation and top-set disagreement (including ties) as descriptive diagnostics.

Second Path Paired with VVWithin-Question Rank Correlation with VVTop-Set Disagreement with VVFusion with VV Minus VV Alone, Public Mean, pp
RR, reference-conditioned0.5950.0%+0.09
VPV_P, paraphrased0.9414.1%−0.21
VZV_Z, higher resolution0.8822.7%+0.01

Reference-free views largely repeat VV's candidate ranking and do not improve accuracy, whereas the reference-conditioned path contributes a distinct judgment. At the same time, sharing the reference couples the failures of the two paths. Across the 98 Value Position pairs where RR fails under the aligned false reference, VEV_E also fails on 79 pairs, whereas VV fails on only 2. These results will be added to the main text and appendix.

AuthorsOpenReview ↗

Response to Reviewer r5LK, Points 2–3/5

2. Practical benefit of retaining the reference-conditioned path

Retaining the reference path improves selection accuracy when the extracted reference is correct, while separating reference access prevents shared error propagation when the reference is wrong. In the submission, we showed component complementarity in Section 4.3 and preserved benchmark selection in Section 4.5, where the five-benchmark mean changes by only 0.03 points while the aligned-reference drop falls from 26.67 to 4.17 pp. Below, we present three new selector-level analyses of the reference path.

First, the reference path adds statistically significant accuracy whenever the extracted reference is correct. In paired comparisons (with full confidence intervals given in Point 3), RefSplit exceeds VV by 1.91 pp on the 711 controlled questions with a correct reference and by 0.67 pp across 5,265 natural candidate pools from ordinary generation on ChartX and ChartBench, where RR alone exceeds VV by 1.11 pp. The accuracy cost is confined to wrong references, where VV performs better, as reported in Section 4.3.

Second, correcting the reference value yields a large accuracy gain over VV on wrong-reference pools. In an oracle analysis on the 2,509 incorrect-reference value-error pools of Section 4.4, we replaced only the value field of the extracted reference with the ground-truth annotated value and re-scored. RefSplit then rises from 25.67% to 31.47%, exceeding VV (26.13%) by 5.34 pp. Because the annotated value coincides with the answer in 1,375 of these 2,509 pools, we interpret this result as an upper bound rather than the gain expected from a modestly better extractor, showing that the reference branch carries information that VV does not recover from the chart alone.

Third, spending the same compute budget on additional reference-free verification does not match RefSplit. When compared against the two reference-free controls from Point 1 (fusing VV with a paraphrased prompt or a 1.5× resolution view), RefSplit beats both controls across all 938 controlled questions by 1.46 and 1.53 pp, both statistically significant. On the public mean, RefSplit exceeds the paraphrased control by 0.30 pp (significant) and is 0.08 pp above the higher-resolution control (not significant), while neither control improves over VV alone.

Finally, performing on par with the non-separated R+VER+V_E on unmodified benchmarks (within 0.03 points) is expected. The two selectors perform almost identically on standard benchmarks and use the same 17 calls, yet under an aligned false reference RefSplit reduces preference loss by 22.50 pp, reduces natural aligned-error selection by 8.43 pp, and improves accuracy across all 2,509 incorrect-reference value-error pools by 0.83 pp. These analyses will be added to the appendix.

3. Statistical significance and computational cost relative to the reference-free baseline

In the submission, Table 10 of Appendix C.1 reported only point differences on the public mean, noting that they are not evidence of statistical equivalence, and did not report wall-clock time against VV. We computed paired bootstrap confidence intervals and measured wall-clock latency for all selectors.

For statistical significance, we used a paired question-cluster bootstrap with 10,000 resamples, in which the same resampled questions are shared across all selectors and candidate policies.

ContrastDifference, pp95% IntervalSignificant at 95%
RefSplit − VV, public mean+0.09[−0.15, +0.34]no
RefSplit − (R+VER+V_E), public mean−0.03[−0.19, +0.13]no
RefSplit − VV, 5,265 natural pools with a correct reference+0.67[+0.22, +1.11]yes
RefSplit − VV, controlled set, 711 questions with a correct reference+1.91[+0.84, +3.05]yes
RefSplit − VV, controlled set, 227 questions with a wrong reference, Section 4.3−6.02[−9.91, −2.31]yes
RefSplit − (R+VER+V_E), aligned-error selection in 664 natural aligned pools−8.43[−10.70, −6.19]yes

On unmodified public benchmarks, RefSplit, VV, and the non-separated selector are therefore statistically indistinguishable, confirming that separation preserves ordinary selection quality (Section 4.5).

For computational cost, we timed the selectors on one A100 GPU with batch size 8 across 128 ChartX questions over three repeats.

SelectorCalls per QuestionSeconds per QuestionRelative to VV
VV82.481.00×
R+VER+V_E175.512.22×
RefSplit175.522.22×

RefSplit and the non-separated selector cost the same, with a ratio of 1.003, so separation itself adds no overhead. Both tables will be added to the appendix.

AuthorsOpenReview ↗

Response to Reviewer r5LK, Points 4–5/5

4. Sensitivity to the fusion weight

RefSplit is not sensitive to the fusion weight. Public benchmark accuracy barely moves across a wide range of weights, and reference separation helps at every weight. In Section 3.4, we set λ=1\lambda = 1 in all primary results. To measure sensitivity directly, we evaluated a pre-specified grid of six weights (λ∈{0,0.25,0.5,1,2,4}\lambda \in \lbrace 0, 0.25, 0.5, 1, 2, 4 \rbrace, where λ=0\lambda = 0 reduces to RR alone). Below, we show the five fused weights, with each cell reporting RefSplit / R+VER+V_E at the same λ\lambda.

Metricλ=0.25\lambda = 0.25λ=0.5\lambda = 0.5λ=1\lambda = 1, reportedλ=2\lambda = 2λ=4\lambda = 4
Public mean, %52.82 / 52.8852.81 / 52.9153.02 / 53.0552.99 / 53.1352.92 / 53.11
Steering, Value Position, pp−19.17 / −27.83−12.83 / −26.83−4.17 / −26.67−0.83 / −25.00−0.50 / −25.50
Steering, Legend Binding, pp−5.33 / −9.33−3.00 / −8.33−0.33 / −7.500.00 / −6.830.00 / −6.00
Controlled set, wrong reference, %52.24 / 50.5153.78 / 51.4056.42 / 53.6057.97 / 56.1361.49 / 56.46
Controlled set, correct reference, %97.13 / 98.0797.13 / 98.0797.13 / 98.1496.78 / 98.3596.35 / 98.28
Natural aligned-error selection, %74.49 / 77.4272.98 / 77.8769.44 / 77.8766.05 / 78.6262.14 / 79.23

Across λ\lambda from 0.5 to 2, RefSplit's public mean varies by only 0.21 points, and no weight gives a statistically significant gain over λ=1\lambda = 1. At the same time, the separation effect is positive and statistically significant at all five weights on both intervention axes (+8.67 to +25.00 pp on Value Position and +4.00 to +7.17 pp on Legend Binding), and RefSplit selects fewer naturally occurring aligned errors than the non-separated selector at every weight. The conclusions of the paper therefore do not depend on choosing λ=1\lambda = 1.

5. Whether a fixed weight undermines the benefits of separation

A fixed weight does not undermine separation. On the contrary, reference separation is what allows the weight to control anchoring, as the grid above illustrates. In Section 4.3, we kept the weight fixed across conditions so that the gain on wrong references would not rely on condition-specific tuning.

First, re-weighting cannot substitute for separation. In the non-separated selector R+VER+V_E, both paths read the shared reference, so no choice of λ\lambda removes the steering, which stays between −25.00 and −27.83 pp across the entire grid. Once reference access is separated in RefSplit, λ\lambda becomes an effective control. As λ\lambda increases from 0.25 to 4, Value Position steering falls from −19.17 to −0.50 pp and wrong-reference accuracy rises from 52.24% to 61.49% (compared with 56.46% for the non-separated selector at λ=4\lambda = 4).

Second, an adaptive weight would need to know at inference time whether the extracted reference is correct, which is not directly observable. In a new analysis on the five public benchmarks, we tested five inference-time signals (including VV's top margin, candidate agreement with the reference, and the verifier's own check of the reference against the chart) as rules for choosing between RefSplit and VV. None of these signals reliably predicts when RefSplit beats VV, so they do not support an adaptive routing rule. We therefore retain the fixed weight λ=1\lambda = 1 and will report the full grid in the appendix, showing that a larger λ\lambda exchanges a small drop on correct references (97.13% to 96.35%) for higher accuracy on wrong ones (56.42% to 61.49%).

We have aimed to answer each question with direct measurements, and we are happy to provide any further analysis that would help the assessment.

Reviewer_1f8LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The current method first generates multiple complete candidate reasoning trajectories. It then extracts a shared reference once, without examining the candidates. Each candidate is subsequently evaluated twice: once with the shared reference as additional context and once without it. The results of these two evaluations are then combined to produce a final ranking.

The paper's contribution lies in demonstrating, through reference intervention experiments using the same backbone, analyses of naturally occurring errors, and experiments across five benchmarks, that the proposed method can reduce such errors while largely preserving standard accuracy.

Critical strengths

The paper is well positioned to investigate how errors propagate through shared references and to validate a targeted mitigation strategy.

Critical weaknesses

The paper does not rule out the possibility that the performance improvements over the baselines are attributable to the capabilities of the underlying foundation model rather than the proposed method. On public benchmarks, RefSplit outperforms V, which directly examines the image, by only 0.09 percentage points, while requiring additional reference extraction and a separate verification path (Appendix C.1). On the controlled subset with incorrect references, RefSplit achieves an accuracy of 56.42%, which is even lower than V’s 62.44% (§4.3). Therefore, while the paper convincingly demonstrates that shared erroneous references can be harmful, it does not sufficiently justify why the reference branch should still be retained.

Questions for authors

The improvements over external baselines may be influenced by differences in the underlying foundation models and chart-related training data. Please clearly specify the baseline checkpoints and provide comparisons using the same backbone under matched training conditions to disentangle the contributions of the model's inherent capabilities from those of the proposed method. I would appreciate it if the authors could further clarify under what conditions RefSplit can achieve consistent and significant performance improvements over directly using V, sufficient to justify its additional computational cost. Furthermore, the paper explicitly instructs R to treat the values and their corresponding bindings in the reference as given facts, but the appendix does not seem to clearly indicate whether VE has been adequately trained on incorrect references. If the verifier were required to actively check the correctness of reference information and were additionally trained on incorrect references, would the reference anchoring effect still persist? Would RefSplit still maintain its advantages over directly using V?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1f8L, Overview and Point 1/5

We sincerely thank the reviewer for the positive assessment and detailed questions, which helped us strengthen the empirical evidence in our paper. Below, we present new analyses and experiments from the discussion phase, all of which will be added to the main text and appendix by the end of this phase.

We address the reviewer's questions across five points (Point 1 below, Points 2–3 in the second comment, and Points 4–5 in the third). If we have overlooked anything, please let us know.

  • Point 1. Which baseline checkpoints were used, and do the gains come from the backbone
  • Point 2. Why the reference branch is retained, and when RefSplit improves over VV
  • Point 3. Whether the additional computational cost is justified
  • Point 4. How RR and VEV_E are instructed, and whether VEV_E was trained on incorrect references
  • Point 5. Whether anchoring persists when the verifier is told to check the reference

1. Which baseline checkpoints were used, and do the gains come from the backbone

In the submission, the comparison that isolates the method is the matched control in Section 4.2, where R+VER+V_E and RefSplit use the same checkpoint, training data, candidate pools, and 17 calls, differing only in whether the grounding path receives the reference. Because the submission did not list the external baseline checkpoints and tested the intervention on only one checkpoint, we will add those model names and have repeated the intervention across additional training runs and model families.

For the external baselines in Table 1, VisualPRM is OpenGVLab/VisualPRM-8B-v1_1 and Perceval is RUC-AIBOX/Perceval-3B-reward. Our verifier is fine-tuned from Qwen3-VL-8B-Instruct. All of our internal selectors run on this single checkpoint, sharing the backbone, the frozen candidate pools, and the training data in Appendix B.2, which consist of ChartQA training examples and re-rendered PlotQA charts.

SelectorVerifier SetupFive-Benchmark Avg.
SCnone52.71%
VVours, no reference52.93%
R+VER+V_Eours, reference given to both paths53.05%
RefSplitours, reference given to RR only53.02%
VisualPRM / Percevalas released51.38% / 50.76%

While Table 1 compares off-the-shelf verifiers across different backbones and training sets, holding the backbone fixed puts VV, R+VER+V_E, and RefSplit within 0.12 points of each other on unmodified benchmarks. When an aligned false reference is introduced, however, preference accuracy drops by 26.67 pp for R+VER+V_E compared with only 4.17 pp for RefSplit.

To check whether this result depends on a single checkpoint or model family, we repeated the Section 4.2 intervention on the same 300 Value Position and 300 Legend Binding pairs across six additional checkpoints of our backbone and three external model families used off-the-shelf without any of our training. Those six checkpoints consist of the untrained base model and five new fine-tuning runs (seeds 0 to 4) where the ranking term was trained without a reference line.

VerifierVEV_E Steering, Value PositionSeparation Effect, Value PositionSeparation Effect, Legend Binding
Submitted checkpoint−24.50+22.50+7.17
Base model Qwen3-VL-8B, no training−51.50+18.33+24.83
New fine-tuning, seed 0−35.67+31.83+9.00
New fine-tuning, seed 1−39.83+33.00+12.00
New fine-tuning, seed 2−40.83+34.00+11.67
New fine-tuning, seed 3−37.33+17.67+9.17
New fine-tuning, seed 4−33.67+25.17+6.00
Qwen2.5-VL-7B, no training−43.67+26.00+28.17
Gemma-4-12B, no training−35.33+27.50+31.17
InternVL3.5-8B, no training−63.17+17.83+26.67

Here, steering is the change in pair preference accuracy (in pp) when an unrelated reference is replaced by an aligned false one. The separation effect is the steering of RefSplit minus that of R+VER+V_E, so a positive value means that separation reduces steering. Every separation effect is positive, and those of the five new fine-tuning runs and the three other model families remain statistically significant after Bonferroni correction. Anchoring therefore occurs even before fine-tuning and across all three external model families (which we evaluate here to test the mechanism rather than as deployed selectors). These results will be added to the appendix.

AuthorsOpenReview ↗

Response to Reviewer 1f8L, Points 2–3/5

2. Why the reference branch is retained, and when RefSplit improves over VV

We retain the reference branch because it provides structured visual premises that direct chart verification VV often misses. As a result, RefSplit is significantly more accurate than VV whenever the extracted reference is correct, while reference separation limits what is lost when the reference is wrong. Section 4.3 showed this using component accuracies, where VEV_E reaches 98.45% compared with 95.22% for VV on correct references and VV reaches 62.44% compared with 49.67% for RR on wrong references. Because that comparison relied on component scores and point estimates, we ran new paired tests of the full selectors on the controlled set (Section 4.3) and on the audited natural candidate pools from ChartX and ChartBench (Section 4.4).

PopulationSizeRefSplit − VV, ppSignificant at 95%
Controlled set, reference correct711 questions+1.91yes
Natural pools, reference correct5,265 pools+0.67yes
Natural pools, reference wrong, no candidate repeats its error1,845 pools+0.28no
Natural pools, reference wrong, a candidate repeats its error664 pools−2.51yes
Controlled set, reference wrong, cited by the reviewer227 questions−6.02yes
Five public benchmarks, mean5,691 questions × 5 policies+0.09no

The gain over VV is consistent and statistically significant when the extracted reference is correct, both on the controlled set and in natural candidate pools. When the reference is wrong, VV is better, as the reviewer notes. In natural pools, that loss is confined to the 664 pools where an incorrect candidate repeats the reference error, matching the exact failure mode our paper diagnoses. With the current extractor, the two regimes offset each other on the public benchmark mean.

To measure how much signal the reference branch carries when the extracted value is accurate, we ran an oracle analysis on all 2,509 incorrect-reference value-error pools in Section 4.4. We replaced only the value field of the extracted reference with the ground-truth annotated value and re-scored the reference-conditioned paths while leaving VV unchanged. RefSplit rises from 25.67% to 31.47%, beating VV (unchanged at 26.13%) by 5.34 pp, while selection of the aligned wrong candidate falls from 69.44% to 45.00%. Because the annotated value coincides with the final answer in 1,375 of the 2,509 pools, we treat this as an upper bound rather than the gain expected from a modestly better extractor. Even so, it shows that the reference branch carries grounding information that VV cannot recover from the chart alone.

These analyses will be added to the appendix.

3. Whether the additional computational cost is justified

Relative to the non-separated reference-conditioned design R+VER+V_E, RefSplit adds no verifier model, training, or verification calls, as both use 17 calls per question. Relative to VV alone, which uses 8 calls, RefSplit is more expensive. Because the submission did not report wall-clock cost against VV, we timed the selectors on one A100 GPU with batch size 8 across 128 ChartX questions over three repeats.

SelectorVerifier Calls per QuestionSeconds per QuestionRelative to VV
VV82.481.00×
R+VER+V_E175.512.22×
RefSplit17 (1 extraction, 16 verifications)5.522.22×

To test whether VV can match RefSplit under a comparable compute budget, we built two 16-call reference-free controls that pair VV with a second reference-free view. The first pairs VV with a paraphrased prompt (2.01× the latency of VV), and the second pairs VV with a 1.5× image-resolution view (2.38× the latency of VV, within 10% of RefSplit).

RefSplit Minus the Reference-Free Control, ppParaphrased Second ViewHigher-Resolution Second View
Controlled set, all 938 questions+1.46, significant+1.53, significant
Five public benchmarks, mean+0.30, significant+0.08, not significant

Neither two-view control improves over VV alone, confirming that extra reference-free compute cannot replace the reference path. For a pipeline that uses no reference at all, VV alone remains a sound low-cost choice. For verifiers that condition on an extracted reference, RefSplit adds no cost over the non-separated design, cuts the aligned-reference drop from 26.67 to 4.17 pp, reduces natural aligned-error selection from 77.87% to 69.44%, and significantly outperforms VV when the reference is correct.

AuthorsOpenReview ↗

Response to Reviewer 1f8L, Points 4–5/5

4. How RR and VEV_E are instructed, and whether VEV_E was trained on incorrect references

VEV_E is never instructed to trust the reference, nor was it trained on candidate-aligned incorrect references. While Section 2.2 and Appendix B.2 describe the prompts and training mixture, we audited the training files to clarify exactly what each role saw during training and will add the results to the appendix.

In our training data, only RR is trained with the instruction to treat the reference as given. The grounding role is trained with 4,513 chart-support negatives built from a wrong cell, a perturbed value, or a swapped series, and all of these are shown as standalone premise checks without a candidate trajectory. No example in the format of VEV_E pairs a false reference with a candidate trajectory. For the 8,353 preference pairs used to train RR, we compared each extracted reference against the ChartQA ground-truth table at a 1% relative tolerance. Of the 6,683 pairs with model-extracted references, 4,043 could be checked against the table, and 54 of these incidentally contain a false reference matching only the rejected candidate. Among the remaining 2,640 pairs where table checking was not applicable, 197 match only the rejected candidate. None of these was constructed as a false-reference training task.

Furthermore, the instruction given to RR does not explain why VEV_E is anchored. Unlike RR, VEV_E uses the exact same chart-grounding instruction as VV, with the extracted record simply added to the input. Even so, its preference accuracy falls from 96.50% to 72.00% under an aligned false reference, whereas VV remains at 97.17%, as shown in Table 2. The anchoring therefore arises from access to the reference rather than from an instruction to trust it.

5. Whether anchoring persists when the verifier is told to check the reference

The anchoring persists even when the verifier is explicitly told to check the reference against the chart, and separation remains useful because it removes the dependency by construction. Because the submission tested VEV_E only with the standard grounding prompt, we ran two new experiments to test reference checking directly and analyzed our trained checkpoints to assess the effect of training.

In the first experiment, we prepended the following verification instruction to the VEV_E prompt.

"The record was extracted automatically and may contain errors. First check each field of the record against the chart and use the record only where the chart supports it."

We then repeated the intervention of Section 4.2 on Value Position and Legend Binding.

VEV_E Steering, ppValue PositionLegend Binding
Standard prompt−24.50−5.50
With the check instruction−26.83−7.67

In the second experiment, we tested whether the verifier can tell that the record is false when asked directly without any candidate trajectory. On Legend Binding, the same checkpoint assigns a mean grounded probability of 0.012 to 400 false records (which include all 300 aligned false records from our intervention) and 0.963 to 400 true records. Yet when such a false record accompanies a candidate trajectory, VEV_E's preference for the correct trajectory still drops. Thus, the model easily detects the chart-reference conflict in isolation, yet still follows the false record when evaluating a trajectory that repeats the same error. This explains why instructing VEV_E to check the reference does not stop the anchoring.

Regarding training on incorrect references, we do not yet have a verifier trained on candidate-aligned false references under the paper's full-model training recipe, so we evaluate the effect of training from the checkpoints in Point 1. Standard grounding training reduces anchoring without removing it. On Value Position, VEV_E steering shrinks from −51.50 pp on the untrained base model to between −24.50 and −40.83 pp across our six fine-tuned checkpoints, none of which was trained on candidate-aligned false references. Training directly on candidate-aligned false references could reduce this sensitivity further, but it requires constructing such errors at training time and provides no guarantee at test time. Withholding the reference from VV gives that guarantee by construction, keeping VV's steering at 0.00 pp on every checkpoint. We therefore view such training as complementary to reference separation. As shown in Point 2, whether RefSplit outperforms VV depends on reference correctness, whereas its robustness gain over the non-separated design holds across every checkpoint and model family we tested. These results will be added to the appendix.

We hope these experiments resolve the reviewer's questions, and we would be glad to run any further analysis during the discussion phase.

Thank you for your response and the additional experiments. I have noted the updates and appreciate your efforts in addressing my comments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript Uploaded

We sincerely thank all reviewers and the Area Chair for the time and effort they have devoted to carefully evaluating our work and providing constructive feedback.

We have uploaded the revised manuscript, incorporating the additional experiments, analyses, and clarifications discussed in our responses. All revisions to the original manuscript are highlighted in blue for ease of review.

For Reviewer 1f8L, we expanded the evaluation across checkpoints and model families, clarified the training conditions and limitations, and strengthened the analyses of reference correctness, statistical significance, and computational cost.

For Reviewer r5LK, we clarified our technical contribution through alignment-specific interventions and additional verification controls. We also added analyses of fusion-weight sensitivity, alternative verification designs, and the benefits and limitations of reference separation.

For Reviewer 7yjw, we provided further evidence for the benefits of retaining a reference-conditioned path and the necessity of separating reference access. We also improved the presentation through clearer notation, comparison tables, and explanations.

The revised manuscript preserves our core method and findings while providing more comprehensive evidence and a clearer discussion of its limitations.

We hope these revisions address the reviewers' concerns and clarify the contribution of our work. We would greatly appreciate your consideration of the revised manuscript in your final assessment.

Thank you again for your valuable time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.