Shared-Reference Anchoring and Reference Separation for Chart Reasoning Verification

Reliable multimodal reasoning verification requires judgments to remain grounded in the source image even when intermediate references are imperfect. Existing verifiers increasingly use model-generated references to make visual premises explicit, but reusing the same fallible reference across multiple verification paths can couple their errors and redirect otherwise distinct judgments toward the same false premise. We identify this failure as shared-reference anchoring and examine its causal effect by varying only the reference while holding the chart, question, and candidate trajectories fixed. To mitigate this dependency, we propose RefSplit, a reference-separated verification method that retains reference-conditioned reasoning while withholding the shared reference from a complementary chart-grounding path using the same verifier checkpoint. On 300 Value Position pairs and 300 Legend Binding pairs, RefSplit reduces the preference-accuracy drop under aligned false references from 26.67 to 4.17 points and from 7.50 to 0.33 points, respectively. The same failure also arises under ordinary generation, where 26.46% of 2,509 audited incorrect-reference value-error pools contain an incorrect candidate aligned with the false reference, and RefSplit reduces aligned-error selection by 8.43 points. Across five public chart benchmarks and five candidate policies, RefSplit achieves 53.02% mean Best-of-8 accuracy, the highest among the evaluated prior selectors and within 0.03 points of the matched non-separated control. These results demonstrate that access to the source image alone does not eliminate the influence of a shared fallible reference. By separating reference access across complementary verification paths, RefSplit limits shared error propagation while preserving candidate-selection quality.