- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates Shared-Reference Anchoring in multimodal chart reasoning verification, where multiple verification paths rely on the same erroneous reference, leading to correlated incorrect judgments. The authors propose RefSplit, which mitigates error propagation by allowing one verification path to use structured reference information while the other directly verifies reasoning against the source chart without access to the shared reference.
Critical strengths
- The paper investigates the correlated judgment biases caused by multiple verification paths sharing the same erroneous reference, addressing a meaningful problem in improving the reliability of multimodal reasoning verification.
- The authors find that incorrect textual references can significantly influence verification judgments even when the verifier still has access to the original chart, which is an interesting observation.
- Furthermore, RefSplit effectively mitigates the influence of erroneous references without introducing additional verifier models or increasing the number of verification calls, while maintaining comparable performance to the non-separated baseline on standard benchmarks.
Critical weaknesses
- The core design of RefSplit simply removes the shared reference from one verification path, allowing it to evaluate reasoning directly against the original chart. Although the authors demonstrate the effectiveness of this strategy in specific error scenarios, further evidence is needed to establish the additional benefits of RefSplit over direct visual verification alone, as well as the necessity of its dual-path design.
- Moreover, the observation that multiple verification paths can exhibit correlated biases when relying on the same erroneous reference, and that removing this reference can mitigate such biases, is relatively intuitive. Therefore, the novelty and technical depth of the proposed approach appear limited.
Questions for authors
Same as weakness.
4 comments
Response to Reviewer 7yjw, Point 2/4
2. Necessity of the dual-path design
Two paths are needed because a single path is either steered by reference errors or blind to structured reference information, and separated access is needed because two paths that share an erroneous reference fail together. In Section 4.2, we showed that an aligned false reference lowers preference accuracy by 26.67 pp when both paths receive it and by 4.17 pp under separated access at the exact same 17 calls. To test both requirements directly, we ran three new analyses (and will add a concise summary of this design to Section 3).
First, we compared one-path and two-path selectors with and without the reference on our submitted checkpoint, including a new separated variant in which reads the reference in place of while remains reference-free.
| Selector | Paths Reading the Reference | Public Mean | Controlled Set, All | Steering, Value Position |
|---|---|---|---|---|
| none | 52.93% | 87.29% | 0.00 pp | |
| alone | its only path | 53.28% | 88.43% | −24.50 pp |
| both | 53.05% | 87.36% | −26.67 pp | |
| RefSplit, | one of two | 53.02% | 87.28% | −4.17 pp |
| , new variant | one of two | 53.09% | 88.99% | −1.67 pp |
Each simpler alternative falls short on one of these two goals. Using alone works well on clean references but loses 24.50 pp under an aligned false reference. Using alone avoids steering but loses the benefit of correct references (−1.91 pp compared with RefSplit and −2.92 pp compared with ). Sharing the reference across two paths in still suffers a 26.67 pp drop. Only the two-path designs with separated access retain the reference while staying robust. Notably, the variant matches RefSplit on the public mean, is 1.01 pp more accurate on correct references in the controlled set, and exceeds across the entire controlled set by 1.71 pp. We will report this variant in the appendix as additional evidence that the benefit extends beyond the specific choice of the reference-reading path and follows from the separation principle itself.
Second, we measured how sharing a reference couples failures across the two paths. Under the aligned false reference, fails on 98 of the 300 Value Position pairs. , which reads the same false reference, fails on 79 of those 98 pairs, whereas , which does not read the reference, fails on only 2 of the 98. A second verification judgment can correct the first only if it does not fail on the same examples.
Third, we verified that this dual-path separation benefit holds across fusion weights, checkpoints, model families, and naturally occurring errors.
| Robustness Check | Result |
|---|---|
| Fusion weight from 0.25 to 4 | Non-separated steering stays between −25.00 and −27.83 pp, while separated steering falls to −0.50 pp |
| Seven checkpoints of our backbone | Separation effect is positive in all seven, and +17.67 to +34.00 pp on Value Position across five new fine-tuning runs |
Three other model families without our training (Qwen2.5-VL-7B, Gemma-4-12B, InternVL3.5-8B) | Separation effect is +17.83 to +27.50 pp on Value Position and +26.67 to +31.17 pp on Legend Binding |
| Naturally occurring aligned errors, 664 pools | Selection of the aligned wrong candidate falls from 77.87% to 69.44%, and accuracy rises from 4.82% to 6.53% |
| All 2,509 incorrect-reference value-error pools from ordinary generation | Separation raises accuracy by 0.83 pp |
Separating the two paths therefore lets the verifier use an explicit reference while keeping one grounding judgment free from reference errors.
Response to Reviewer 7yjw, Points 3–4/4
3. Whether the finding is intuitive, and novelty and technical depth
We agree that, in hindsight, the advice not to share a faulty input sounds natural. However, intuition alone does not explain why the chart-grounded path , which sees the source chart directly and is instructed to check it, still follows a false text reference. Prompted by the reviewer's comment, we tested four simpler explanations for this behavior, and none of them accounts for the drop.
First, a path that sees the chart still follows the false text across four model families.
| Path, Value Position, 300 Pairs, Table 2 | Unrelated Reference | Aligned False Reference |
|---|---|---|
| , instructed to treat the reference as given | 95.33% | 66.67% |
| , instructed only to check the chart, also receives the reference | 96.50% | 72.00% |
| , never receives the reference | 97.17% | 97.17% |
loses 24.50 pp even though the chart is in front of it and its prompt is the exact same chart-grounding instruction used by . In new experiments, the same collapse occurs without any of our training on three other model families, with steering of −43.67 pp on Qwen2.5-VL-7B, −35.33 pp on Gemma-4-12B, and −63.17 pp on InternVL3.5-8B. Furthermore, adding the reference line to the released external verifier VisualPRM causes it to lose 29.83 pp in the same comparison (falling from 86.33% to 56.50%).
Second, the drop does not happen because the chart is unreadable, as the verifier recognizes that the false record contradicts the chart when asked directly. In a new experiment, we asked the same checkpoint whether standalone records are supported by the chart without any candidate trajectory. Across 400 false records (which include all 300 aligned false records from the intervention) and 400 true records on Legend Binding, the checkpoint assigns a mean grounded probability of 0.012 to false records and 0.963 to true ones. Thus, the verifier easily detects that the record is false in isolation, but when the same record is supplied as context for judging a trajectory, it still lowers 's preference for the correct trajectory.
Third, explicitly instructing the verifier to check the reference against the chart first does not fix the problem. We added a sentence to the prompt warning that the record was extracted automatically and may contain errors and asking the verifier to check each field against the chart before using it. Under this instruction, Value Position steering remains at −26.83 pp, compared with −24.50 pp under the standard prompt.
Fourth, the accuracy drop is driven by agreement with a wrong candidate, not by the plausibility of the reference error. Across 290 Value Position pairs, replacing the aligned false value with another value plotted elsewhere in the same chart (mentioned by neither candidate) leaves accuracy at 94.66%, whereas the candidate-aligned false value drops it to 71.38%. An empty reference line has no detectable effect either (92.33% compared with 92.50%). A reference error is therefore harmful specifically when a candidate makes the same error, which occurs naturally in 26.46% of the 2,509 incorrect-reference value-error pools we audited under ordinary generation.
None of these four outcomes follows from simple intuition. A verifier that sees the chart, recognizes the false record in isolation, and is explicitly told to check the chart still follows the record when a candidate agrees with it. Our technical contribution lies in isolating this directional dependency and removing it through a matched design that holds the checkpoint, prompt interface, candidates, and 17-call budget fixed. These analyses will be added to the appendix.
4. Making the paper easier to follow
We agree that readers currently have to track the three verification paths and their inputs across sections without a single reference table, so we will make four concrete revisions to improve readability:
- A compact notation table at the start of Section 2 will list, for , , and , the inputs each path receives, its instruction, and its role, alongside the four reference conditions.
- Section 3 will open with a three-sentence summary of the design before introducing the equations.
- Appendix C will report and alongside RefSplit with paired bootstrap tests in a unified table so that all selector comparisons can be read in one place.
- Each main-text section will include explicit pointers to the supporting analyses in the appendix.
We would be grateful for pointers to any passage that remains hard to follow.
Response to Author
Hi, apologies for my late response.
Thank you for your response and the additional experiments. While the new results address some of my concerns, I still have reservations about the novelty of RefSplit, as its core design appears relatively straightforward. I am not yet fully convinced that the proposed approach offers sufficient methodological novelty beyond separating reference access between two verification paths.
Response to Reviewer 7yjw, Overview and Point 1/4
We sincerely thank the reviewer for the thoughtful summary and for noting that RefSplit adds no verifier model and no verification calls over the non-separated baseline. The review helped us make the motivation and evidence for our design much more explicit. For each comment raised by the reviewer, we present new analyses and experiments carried out during the discussion phase, all of which will be added to the main text and appendix by the end of this phase.
We address the reviewer's comments across four points (Point 1 below, Point 2 in the second comment, and Points 3–4 in the third). If we have overlooked anything, we would be grateful if the reviewer could let us know.
1. Additional benefit of RefSplit over direct visual verification alone
We agree that V is a strong baseline, and the paper does not claim that RefSplit beats V on the unstratified public average, where the +0.09 pp difference is not statistically significant. Rather, the reference path adds accuracy when the extracted reference is correct. Because Section 4.3 showed this using component accuracies on the controlled set, we evaluated the full selector directly on both the controlled set and candidate pools from ordinary generation, as well as against compute-matched reference-free controls.
The gain over V is therefore present in ordinary generation as well as on the controlled set, and it cannot be reproduced by spending the same compute budget on additional reference-free verification. At the same time, the trade-off is clear. When the reference is wrong, V is more accurate (Section 4.3), and RefSplit requires 17 calls compared with 8 for V (5.52 s per question against 2.48 s, or 2.22×). RefSplit is therefore designed for pipelines that use an extracted reference. In Point 2, we show why such pipelines require two separated paths. These analyses will be added to the appendix.