acceptodds
Under review as a conference paper at ICLR 2027

Beyond Generation and Accuracy: Diagnosing and Enhancing Visual Chain-of-Thought for Geometry Problem Solving

Abstract

Solving geometric problems often requires more than passively reading a given diagram; constructing auxiliary lines is crucial to reveal hidden geometric relations. While Visual Chain-of-Thought (VCoT) allows models to introduce such intermediate visual aids, a key question remains: why do autonomous systems struggle to benefit from them? In this work, we present the first systematic study that decouples the potential value of visual aids from a model’s ability to construct and effectively use them. Specifically, we evaluate models under three controlled setups—No-Aux, Auto-Aux, and GT-Aux—along with a five-dimensional trajectory diagnosis on our proposed benchmark, GeoVAD-Bench. Our analysis reveals a critical bottleneck in VCoT, which we call the autonomy gap: while high-quality auxiliary diagrams consistently boost accuracy, autonomous models fail to capture these gains when generating aids on their own. Diagnostic results show that errors compound across perception, construction, utilization, and deduction, which ex- plains why visual quality alone cannot account for this performance gap. Driven by these insights, we introduce a targeted post-training paradigm and the first reinforcement learning algorithm designed for interleaved visual-textual reasoning. Our model, GeoWeave-8B, outperforms the base model by 25.3 percentage points in final geometric accuracy and 30.4 percentage points in process diagnostic metrics. These results demonstrate that a diagnosis-guided approach provides a clear path toward closing the autonomy gap in geometric reasoning. Code is available in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Unified Multimodal Model, Visual Chain-of-Thought, Geometry Problem Solving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3GubOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Visual Chain-of-Thought for geometry problem solving, distinguishing the potential value of high-quality auxiliary diagrams, a model's ability to construct them autonomously, and their use in subsequent reasoning. The authors introduce GeoVAD-Bench, which contains 600 problems, and define an autonomy gap through three evaluation conditions: No-Aux, Auto-Aux, and GT-Aux. Model trajectories are evaluated along five dimensions: original-diagram perception, auxiliary-diagram quality, auxiliary-diagram utilization, reasoning correctness, and final-answer correctness.

Building on this diagnosis, the authors perform progressive supervised fine-tuning of SenseNova-U1-8B using geometric perception, diagram editing, and interleaved visual-textual reasoning data, followed by Interleave-RL with step-level process feedback. The resulting model, GeoWeave-8B, improves answer accuracy on GeoVAD-Bench from 37.3% to 62.6% and Process Avg. from 52.5% to 82.9%. The paper also reports results on several public benchmarks.

Critical strengths
  1. Distinguishing whether auxiliary diagrams are useful from whether models can autonomously realize their benefits is a valuable research direction. In Table 1, GT-Aux improves accuracy over No-Aux by 3.0–7.0 percentage points across the three systems, whereas Auto-Aux underperforms No-Aux for the two smaller models. This comparison shows that evaluating autonomous drawing systems solely by final-answer accuracy can obscure the potential value of valid auxiliary constructions.
  2. The process evaluation covers multiple stages before and after diagram construction, beyond final-answer correctness or generated-image quality. In particular, separately evaluating auxiliary-diagram quality and utilization helps distinguish incorrect construction from ineffective use of a constructed diagram. Table 2 further shows that the gap between correct and incorrect solutions is larger for reasoning-process scores than for diagram-quality scores, supporting attention to both construction and subsequent reasoning.
  3. The training ablations on the same base model provide positive evidence for the overall approach. In Table 5, SFT raises answer accuracy to 51.6%, reinforcement learning without Step-level Credit Assignment (SCA) further improves it to 60.3%, and the full method reaches 62.6%. SCA also improves reasoning-process accuracy from 46.3% to 66.3%. These results support the value of process feedback in this training setting.
Critical weaknesses
  1. The central claim of narrowing the autonomy gap lacks a direct evaluation consistent with its definition. Equation (2) defines this gap as the difference between GT-Aux and Auto-Aux accuracy for the same model, but Tables 3 and 5 do not report GT-Aux results for the trained GeoWeave model, so its post-training gap cannot be calculated. The absence of GeoWeave's No-Aux results also prevents assessing whether training increases the net benefit of autonomous visual operations relative to reasoning without generating auxiliary diagrams. The reported answer and process metrics support improvements in the overall system, but do not yet distinguish improvements in general geometric reasoning from increased benefits of autonomous visual operations. I suggest reporting the corresponding conditions for the final model or qualifying the claims about narrowing the gap to match the scope directly supported by the experiments.
  2. The error-attribution protocol for Figure 2 is insufficiently specified, limiting interpretation of the diagnostic distribution. Section 3.2 uses this distribution to motivate the four main process dimensions, but the main text and Appendix A.4 do not specify how an attribution category is selected when multiple error types occur in the same trajectory, or what cases fall under Others. It is therefore unclear whether the proportions reflect the first error, the dominant error, or another attribution rule. The four categories cover approximately 90% of cases, making them reasonable primary dimensions for analysis. Providing the attribution rule for trajectories with multiple errors and a few examples of Others would clarify the meaning of these diagnostic findings.
Questions for authors
  1. Could you report No-Aux and GT-Aux results for the final GeoWeave model and compare them with the corresponding base-model conditions in Table 1? If a full evaluation is costly, a comparison on the same fixed subset of problems with comparable prompts and budgets would also help assess the post-training realized value and autonomy gap.
  2. To clarify the methodology behind Figure 2, could you briefly explain what cases are included in Others and which attribution rule is used when multiple error types occur in the same trajectory?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3Gub

We thank the reviewer for recognizing the distinction between potential and realized visual-aid value, our multi-stage process diagnosis, and the training ablations. We address your questions below.

W1 / Q1. Direct measurement of the post-training autonomy gap

The controlled-evaluation results across the base model, SFT checkpoint, and final model are:

ModelNo-AuxAuto-AuxGT-AuxVpotentialV_{\mathrm{potential}}VrealizedV_{\mathrm{realized}}GautonomyG_{\mathrm{autonomy}}
SenseNova-U1-8B43.837.350.8+7.0−6.513.5
GeoWeave-8B-SFT45.251.652.5+7.3+6.40.9
GeoWeave-8B57.762.666.3+8.6+4.93.7

GeoWeave-8B narrows GautonomyG_{\mathrm{autonomy}} from 13.5 to 3.7 pp and turns VrealizedV_{\mathrm{realized}} from −6.5 to +4.9 pp. Beyond strengthening general geometric reasoning, training makes autonomous visual assistance beneficial over the same model's No-Aux baseline. The SFT checkpoint shows the same trend.

W2 / Q2. Attribution rule for multiple errors and the meaning of Others

Thank you for noting that Figure 2's attribution protocol needs clarification. We use first-error attribution, and Others primarily covers three failure modes:

TypeDescription
Reasoning loop / non-completionThe trajectory enters repetitive reasoning or fails to complete a solution.
Degenerate generation / garbled outputThe generated output degenerates or becomes unintelligible, preventing a coherent solution.
Partial coverage of a multipart problemThe response addresses only part of a problem containing multiple subquestions, leaving required parts unanswered.

Representative examples: https://anonymous.4open.science/r/GeoWeave-05CC/others_showcase.pdf.

First-error attribution. Each failed trajectory is assigned to its first identifiable error or failure, avoiding ambiguity from downstream errors that stem from an earlier mistake. We will clarify both this rule and the definition of Others in the revised manuscript.

We have incorporated the key experiments suggested by the reviewers into the revised PDF (pages 9–10), with all additions highlighted in blue.

Thank you for the additional controlled evaluations and the clarification of the error-attribution protocol. The new results address my question about the post-training autonomy gap, and the explanation of first-error attribution and Others resolves my second clarification request. I will hold on to the current rating.

Thank you for your follow-up. We are glad that our additional evaluations and clarifications have addressed your questions. We appreciate your time and thoughtful feedback.

Reviewer_uLtpOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the gap between a model's ability to generate visual aids and its ability to actually use them effectively during geometric reasoning, introducing the concept of an "autonomy gap." The authors create GeoVAD-Bench and a five-dimensional diagnostic framework to show that while ground-truth auxiliary diagrams significantly improve reasoning, models generating their own diagrams often fail due to compounding errors in perception, construction, utilization, and deduction. To bridge this gap, they propose GeoWeave, a training framework combining capability-aligned supervised fine-tuning with an interleaved visual-textual reinforcement learning paradigm that uses step-level credit assignment. Their 8B model substantially outperforms baselines on both their benchmark and public mathematical reasoning datasets.

Critical strengths
  1. The "Autonomy Gap" is a Brilliant Conceptualization. The authors perfectly formalize a nagging intuition in the VCoT community: there is a huge difference between a model being able to solve a problem when handed a perfect diagram (GT-Aux), and a model being able to autonomously generate and use that diagram (Auto-Aux). The revelation that Auto-Aux can actually perform worse than No-Aux (e.g., a 10-point drop for Qwen3-VL) is a stunning result. It proves that bad visual aids act as active distractors, not just neutral noise.

  2. The Diagnostic Framework (L1-L5) is Exactly What the Field Needs. The 5-dimensional breakdown (Perception, Quality, Utilization, Reasoning, Answer) elegantly disentangles where the pipeline breaks. The pie charts in Figure 2 showing that Construction Errors and Unused Constructions account for a massive chunk of failures perfectly motivate their later training pipeline. They prove that building a nice-looking diagram is useless if the model doesn't explicitly reference it in the next reasoning step.

  3. Progressive SFT aligns perfectly with the Diagnosis. Instead of throwing all the data into one massive training run, they build the model step-by-step: Perception first, then Diagram Editing, then full Interleaved Reasoning. It's a highly intuitive curriculum learning approach that respects the cognitive complexity of the task.

  4. The RL Design is Pragmatic and Smart. A lot of current multimodal RL papers try to compute policy gradients across the entire continuous pixel-diffusion space, which is notoriously unstable and expensive. GeoWeave's decision to only apply the policy gradient to the text reasoning steps (using Step-Level Credit Assignment, SCA) while keeping the image generation branch stable via velocity-MSE is brilliant. It aligns perfectly with their diagnostic finding: the bottleneck isn't just drawing the line, it's knowing when to draw it and how to use it. The training dynamics in Figure 5 (where the model without SCA collapses into repetition) strongly validate this design choice.

Critical weaknesses
  1. Heavy Reliance on LLM Judges for RL. The Step-Level Credit Assignment relies on a Process Verifier (Gemini-3.5-Flash) to give error weights to each reasoning step. While this works well in geometry (where errors are mathematically objective), it makes the training loop dependent on an expensive, external proprietary model. It also raises the risk of reward hacking—if the verifier makes a mistake or has a bias, the policy will aggressively exploit it.

  2. Single Base Model Architecture. The entire GeoWeave training pipeline is built on a single base model (SenseNova-U1-8B). While the authors test other models in the evaluation (GPT-5.4, Qwen), I would love to see if the SFT + Interleave-RL recipe generalizes to other unified architectures (like Bagel or Janus). Is the success due to the method, or does it require the specific native visual readback architecture of SenseNova?

  3. The Compute Cost of Interleaved Rollouts. Generating multiple images, re-encoding them, and doing 8 rollouts per prompt (as mentioned in Appendix D.2) inside a single training step is computationally brutal. While the paper is scientifically valuable, it is practically quite heavy to reproduce for an academic lab without massive GPU resources.

Questions for authors

First, regarding the Step-Level Credit Assignment, the entire RL loop relies on Gemini-3.5-Flash acting as the process verifier to assign error weights to each reasoning step. Could you clarify how robust this verifier is to domain-specific but unconventional reasoning? If the external verifier incorrectly flags a valid geometric derivation as an error, the policy will be penalized for correct behavior. Did you conduct a human audit on a subset of the verifier's step-level annotations to establish its agreement rate with human experts? Furthermore, did you observe any instances of reward hacking, where the model learned to format its reasoning to appease the verifier rather than actually improving its mathematical logic?

Second, the "Auto-Aux" performance in Table 1 drops significantly for Qwen3-VL and SenseNova. I want to understand the exact failure mode in the baseline. When these models perform worse with autonomous generation than without it, is it because they generate a visually incorrect diagram (Construction Error), or because they generate a correct diagram but completely fail to ingest the new visual state into their context (Utilization Error)? If it is the latter, it suggests the autonomy gap is fundamentally a context-window limitation rather than a reasoning limitation, which would slightly shift the narrative of the paper.

As a future direction, the training pipeline is heavily tied to SenseNova-U1-8B, which is a native interleaved model. Since Unified Multimodal Models differ significantly in their native interleaving capabilities and image-readback mechanisms, it would be incredibly valuable to see if this SFT plus Interleave-RL recipe transfers directly to a different architecture, such as Bagel, or a pipeline-based model (like Qwen-VL paired with an external editor). Testing a different backbone would prove whether the success is due to your diagnosis-guided methodology or specific to SenseNova's architecture.

Finally, regarding the scalability of the method, generating multiple images and performing eight rollouts per prompt inside a single training step is computationally heavy. For future work, it would be highly practical to explore distilling the expensive LLM process verifier into a smaller, local reward model. This would drastically lower the wall-clock time and API costs of the Interleave-RL stage, making this excellent methodology much more accessible to academic labs with limited compute.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uLtp

We thank the reviewer for recognizing the autonomy-gap formulation, the trajectory-level diagnostic framework, and the alignment between our progressive supervision and interleaved-RL design. We address your concerns on verifier reliability, architectural generality, and computational cost below.

W1 / Q1. Reliance on an external process verifier and reward hacking

External-verifier dependence. We acknowledge that our current pipeline relies on a proprietary model for process verification. To reduce this dependency, we will explore majority voting across multiple small open-source verifiers and training a compact local verifier on human-audited step-level annotations.

Human agreement for the SCA process verifier. Auditing 800+ reasoning steps against human annotations confirms high verifier reliability:

VerifierAgreementCohen's κGwet's AC1
Gemini97.17%0.83470.9659

To avoid penalizing valid but unconventional derivations, the verifier prompt explicitly excludes differences in method, skipped steps, and writing quality from penalties, requiring an explicit false claim with a verifiable counterargument before flagging an error (Appendix G.2.2).

Checks for reward hacking. Comparing the SCA ablation under the original protocol (Gemini-3.1-Pro for L1–L4; paper protocol for L5) and the independent GPT-5.6-Terra re-evaluation:

ProtocolModelL4Process Avg.L5
OriginalGeoWeave-8B w/o SCA46.376.560.3
OriginalGeoWeave-8B66.382.962.6
GPT-5.6-TerraGeoWeave-8B w/o SCA47.575.159.8
GPT-5.6-TerraGeoWeave-8B63.579.961.8

The independent GPT judge preserves the SCA gains (+16.0 pp in L4, +4.8 pp in Process Avg., and +2.0 pp in L5 vs. +20.0, +6.4, and +2.3 pp originally), confirming that improvements are not confined to Gemini-family judges. Moreover, as shown in enZ4–W3/Q4, SCA increases mean assertions per trajectory from 20.26 to 28.28 (median 17 to 20), arguing against claim suppression. Together, human agreement, cross-judge consistency, and assertion counts show no evidence of reward hacking in these checks.

W2 / Q3. Applicability beyond a single base architecture

Our training data, supervision taxonomy, and Interleave-RL algorithm are architecture-agnostic, with no components tailored to SenseNova-U1. The three supervision categories operate at the task level, and Interleave-RL assigns step-level textual credit while regularizing the image-generation branch. The framework therefore applies in principle to any AR + DiT unified multimodal model (including Bagel), and we plan to add Bagel experiments as computational resources become available to validate cross-backbone applicability.

W3 / Q4. Reducing rollout and verifier costs

Rollout infrastructure. We already mitigate interleaved rollout overhead through three infrastructure optimizations (Appendix C.2):

  • Dynamic rollout scheduling reduces cross-device blocking across trajectories with varying numbers of image-generation steps.
  • Local continuous batching increases inference concurrency.
  • GPU-memory optimizations support stable multi-turn text generation, image generation, and readback.

We will continue integrating advances in multimodal serving to further lower rollout cost.

Verifier cost. To reduce API expenditure and improve reproducibility, we will investigate open-source verifier ensembles and a compact local process verifier (as noted in uLtp–W1/Q1), validating their annotation reliability and policy-optimization behavior before replacing the current verifier.

Q2. Auxiliary-diagram quality and truncation

Poor auxiliary-diagram quality is a primary failure mode for both SenseNova-U1-8B and Qwen3-VL-8B + Qwen-image-edit, as reflected in their low Table 3 L2 scores:

ModelL2: Aux. Quality
SenseNova-U1-8B25.4
Qwen3-VL-8B + Qwen-image-edit21.1

Access to image generation alone does not guarantee faithful constructions; inaccurate diagrams introduce misleading geometric relations into subsequent reasoning, explaining why autonomous drawing can impair performance.

Regarding truncation, SenseNova-U1-8B has a ~10% truncation rate, compared with 50% for Qwen3-VL-8B + Qwen-image-edit. Qualitative inspection indicates that Qwen's truncation primarily occurs when low-quality diagrams create ambiguity between the visual state and geometric deductions, leading to prolonged reasoning. Thus, truncation in these inspected cases is largely a downstream consequence of unreliable visual construction rather than an image-ingestion failure.

Representative visualized examples: https://anonymous.4open.science/r/GeoWeave-05CC/auto_aux_showcase.pdf.

We have incorporated the key experiments suggested by the reviewers into the revised PDF (pages 9–10), with all additions highlighted in blue.

Dear Reviewer uLtp,

Thank you again for your time and thoughtful feedback. We would like to kindly follow up on our response to your comments. We hope that the explanations and additional results have helped address your concerns, and we would be happy to provide any further clarification.

If your schedule permits, we would greatly appreciate any further feedback before the discussion period closes.

Best regards, The Authors

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal Responses

Dear Reviewer uLtp,

We sincerely thank you for your thoughtful review, constructive suggestions, and the time dedicated to evaluating our work.

We have carefully addressed all your concerns with detailed responses, along with additional experiments and analyses in the revised manuscript. As the discussion period is nearing its end, we would greatly appreciate it if you could review our revisions when your schedule permits and let us know if any concerns remain or require further clarification.

Thank you again for your valuable feedback and contribution to the review process. We wish you all the best in your research.

Best regards,

The Authors

Reviewer_enZ4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies visual chain-of-thought (VCoT) for geometry problems that need auxiliary constructions.

Diagnostic framework. Models are evaluated under three controlled conditions: No-Aux, Auto-Aux (the model draws its own auxiliary diagram), and GT-Aux (a validated reference construction is given). From these the paper defines potential value (A_GT − A_No), realized value (A_Auto − A_No), and the autonomy gap (A_GT − A_Auto) (Eq. 2). It adds L1–L5 trajectory diagnostics (perception, auxiliary-diagram quality, utilization, reasoning correctness, answer) and releases GeoVAD-Bench: 600 problems over three difficulty levels, 360 of them in Chinese.

Diagnostic finding. Open-source 8B systems do worse when they draw: Auto-Aux is below No-Aux.

Method. GeoWeave-8B (base: SenseNova-U1-8B) is trained in two parts:

  • Three-stage progressive SFT on about 700K synthetic examples (perception, editing, interleaved solutions), largely generated or rewritten with Gemini-3.5-Flash.
  • Interleave-RL: GRPO on the text branch with CAPO-style step-level credit assignment (SCA) from a Gemini-3.5-Flash process verifier. The image branch gets only a velocity-MSE regularizer, with no policy-gradient update.

Results. On GeoVAD-Bench (Auto-Aux), GeoWeave-8B improves answer accuracy by 25.3pp and Process Avg by 30.4pp over the base model. It also reports gains on MathVista (GPS subset), MathVerse, MATH-Vision, and Math-VR.

Critical strengths

S1. A clean, reusable decomposition of the VCoT question. The No-Aux / Auto-Aux / GT-Aux design and Eq. 2 separate three questions: is the diagram useful, can the model construct it, and is it used? This gives an analysis framework that transfers to other VCoT domains, and it is the paper's main conceptual contribution.

S2. A useful negative finding. Drawing currently hurts open 8B unified models: Qwen3-VL-8B + Qwen-image-edit drops from 43.0 to 36.0, and SenseNova-U1-8B from 43.8 to 37.3 (Table 1). Being able to generate images is not the same as benefiting from them. A drop of 6.5–7.0pp is fairly robust even at n = 600, and the community should know about it.

S3. Practical interleaved-RL infrastructure, plus evidence that step-level credit stabilizes training. The multi-round "text → image generation → readback → text" rollout system supports a variable number of images (up to 4) and uses the same generation configuration in training and inference (App. C.2), which makes it useful engineering for unified-model post-training. Without SCA, response length explodes and training collapses (Figure 5), a useful empirical observation about RL on interleaved trajectories.

Critical weaknesses

W1. The central claim, closing the autonomy gap, is never measured for the proposed model. G_autonomy is defined as A_GT − A_Auto (Eq. 2), but GeoWeave-8B is reported only under Auto-Aux (Tables 3 and 5). Without No-Aux and GT-Aux numbers, it is impossible to tell whether GeoWeave got better at building and using diagrams (the gap shrinks) or simply got better at geometry reasoning overall (A_No and A_GT rise together, and the gap may not shrink at all). The title, abstract, and conclusion all rest on this claim.

W2. The gains are not attributed to the visual aid.

  • No text-only control. There is no same-data text-only control, e.g., the same 100K interleaved solutions with the images removed, followed by the same RL.
  • No image counterfactual. No experiment replaces the generated diagram with the original, a wrong, or a blank diagram.
  • L3 cannot detect real use of the diagram. L3 ("utilization") is scored from the text, and "a non-empty extracted instruction is authoritative" (p.28). L3 therefore cannot tell whether the model used the diagram or only its own written construction instruction.

As things stand, "700K examples of geometry data plus RL on text reasoning" explains Table 3 just as well.

W3. The process metrics share a source with the RL reward and have no human validation.

  • Same family, near-identical criteria. The L1–L4 judge (Gemini-3.1-Pro) and the RL process verifier (Gemini-3.5-Flash) are in the same model family and apply almost the same rule: flag only explicit false claims, and do not flag when uncertain (p.29 vs. p.31).
  • Shared style. The SFT solutions were rewritten by Gemini-3.5-Flash, which is also one of the evaluated systems.
  • An asymmetry consistent with Goodhart. SCA raises Reasoning from 46.3 to 66.3 but accuracy only from 60.3 to 62.6 (Table 5). That pattern fits a model learning to avoid falsifiable assertions just as well as genuinely better reasoning.
  • No validation. There is no human-judge agreement study, yet Process Avg +30.4pp is one of the two headline numbers.
Questions for authors
  1. What are the accuracies of GeoWeave-8B, and its SFT checkpoint, under No-Aux and GT-Aux? Is G_autonomy actually smaller than the base model's 13.5pp, and does V_realized turn positive?
  2. Please add (a) a same-data text-only SFT + RL control evaluated under No-Aux and (b) image counterfactuals: original, wrong, or blank diagram, and a masked construction instruction. How much does accuracy drop when the generated diagram is swapped out?
  3. Where do the 600 GeoVAD-Bench problems come from? Were they deduplicated against the in-house textbook/exam set, MMK12, Geo170K, and the other SFT/RL sources (text near-duplicates, image perceptual hashes, answer matching)? What are the results on a deduplicated subset?
  4. Is there any human-judge agreement data for L1–L4 (κ/α on 200 or more trajectories from several systems)? How do Tables 3 and 5 change under a non-Gemini judge? How does the number of explicit assertions per step change before and after SCA?
4 comments
AuthorsOpenReview ↗

Response to Reviewer enZ4 – Part1/2

We thank the reviewer for recognizing our decomposition of visual-aid value, the finding that autonomous drawing can impair reasoning, and our interleaved-RL framework with step-level credit assignment. We address your suggestions below (all accuracy differences are in percentage points, pp).

W1 / Q1. Direct measurement of the post-training autonomy gap

We agree that Auto-Aux accuracy alone does not establish a reduction in GautonomyG_{\mathrm{autonomy}}, so we report full No-Aux and GT-Aux comparisons below:

ModelNo-AuxAuto-AuxGT-AuxVpotentialV_{\mathrm{potential}}VrealizedV_{\mathrm{realized}}GautonomyG_{\mathrm{autonomy}}
SenseNova-U1-8B43.837.350.8+7.0−6.513.5
GeoWeave-8B-SFT45.251.652.5+7.3+6.40.9
GeoWeave-8B57.762.666.3+8.6+4.93.7

GeoWeave-8B reduces GautonomyG_{\mathrm{autonomy}} from 13.5 to 3.7 pp and turns VrealizedV_{\mathrm{realized}} from −6.5 to +4.9 pp. Thus, training not only strengthens general geometric reasoning (No-Aux: 43.8 → 57.7), but also improves the model's ability to autonomously construct and utilize visual aids during reasoning. The SFT checkpoint shows the same trend relative to the base model.

W2 / Q2. Attribution of improvements to visual assistance

Text-only control. The text-only control uses the same source dataset with only the image segments removed and the text retained. Comparing both training stages:

StageText-onlyInterleavedGain
SFT49.851.6+1.8
RL58.162.6+4.5

Interleaved training outperforms the text-only control by 1.8 pp after SFT and 4.5 pp after RL, showing that visual assistance contributes beyond textual supervision alone, especially after RL.

Counterfactual experiments. We also intervene on the first generated auxiliary image during reasoning:

ConditionAcc.Δ
Control62.3—
first auxiliary image→original54.5−7.8
first auxiliary image→blank55.3−7.0
mask the first construction instruction55.7−6.6

Replacing the generated auxiliary image with the original or a blank image reduces accuracy by 7.8 and 7.0 pp, respectively, while removing its construction instruction reduces accuracy by 6.6 pp. Thus, both the auxiliary diagram and its construction instruction are essential to performance.

What L3 establishes. We agree that L3 is scored primarily from reasoning text and cannot by itself separate reliance on the textual instruction from reliance on the generated image. L2 and L3 are designed to decouple visual-aid quality from intended construction use: L2 measures diagram quality, whereas L3 measures whether the elements and relations introduced by the instruction are integrated into subsequent reasoning (assuming faithful visual aids). While L3 alone does not measure causal dependence on the image, the counterfactual interventions above directly establish the importance of both the image and the instruction.

Continued in Part 2/2 below due to character limit.

AuthorsOpenReview ↗

Response to Reviewer enZ4 — Part 2/2

W3 / Q4. Judge dependence, human validation, and possible reward exploitation

We address the concern of using same-family models for supervision, process rewards, and evaluation through human validation, an independent non-Gemini judge, and an explicit assertion-count analysis.

Human agreement for the diagnostic metrics. We audited 200 trajectories sampled across four models (CodePlot-CoT-32B, SenseNova-U1-8B, MathCanvas-7B, and GeoWeave-8B) against human annotations:

MetricAgreementCohen's κGwet's AC1
L1: Perception86.60%0.47370.8220
L2: Aux. Quality80.28%0.75000.7546
L3: Aux. Util84.29%0.49800.8295
L4: Reasoning93.26%0.86440.8662
L5: Answer95.77%0.90930.9207

Agreement is strong across dimensions, particularly for reasoning-process correctness (L4) and final answers (L5). The lower κ values on L1 and L3 stem from prevalence imbalance (human annotators assign a score of 1 to 79.38% of L1 cases and 82.14% of L3 cases), which depresses Cohen's κ despite high observed agreement. Using prevalence-robust Gwet's AC1 yields 0.8220 (L1) and 0.8295 (L3) alongside 86.60% and 84.29% exact agreement, confirming substantial human–judge alignment.

Evaluation by a non-Gemini judge. Re-evaluating key models from Tables 3 and 5 with GPT-5.6-Terra yields:

ModelL1L2L3L4L5Process Avg.
CodePlot-CoT-32B79.371.586.741.448.469.7
Bagel-Zebra-7B56.72.925.118.525.025.8
SenseNova-U1-8B69.339.374.423.635.351.7
MathCanvas-7B60.367.376.626.047.857.6
GeoWeave-8B-SFT68.578.889.335.651.768.1
GeoWeave-8B w/o SCA77.681.593.747.559.875.1
GeoWeave-8B81.881.093.363.561.879.9

Process Avg. retains the paper's definition as the mean of L1–L4 (L5 here reflects this re-evaluation rather than replacing the paper's answer-evaluation protocol). Although absolute scores shift, GeoWeave-8B remains best in Process Avg., L4, and L5, and SCA still yields clear gains in L4 (47.5 → 63.5, +16.0 pp), Process Avg. (75.1 → 79.9, +4.8 pp), and L5 (59.8 → 61.8, +2.0 pp).

Human agreement for the SCA process verifier. A separate audit of 800+ reasoning steps yields 97.17% exact agreement, Cohen's κ = 0.8347, and Gwet's AC1 = 0.9659 between human annotators and the Gemini process verifier, directly validating the step-level SCA signals.

Explicit assertion counts. To examine whether SCA improves process scores merely by avoiding explicit claims, we count assertions in the generated trajectories:

ModelMean assertions/trajectoryMedianRange
GeoWeave-8B-SFT18.61174–64
GeoWeave-8B w/o SCA20.26174–108
GeoWeave-8B28.28204–105

With SCA, the mean assertion count increases from 20.26 to 28.28 (median 17 to 20). Thus, higher reasoning scores accompany more explicit assertions rather than fewer (reported as trajectory-level counts, not normalized by step count). Combined with human validation and the GPT-5.6-Terra re-evaluation, this argues against simple claim suppression as the driver of L4 gains.

Q3. Benchmark provenance and train–test deduplication

GeoVAD-Bench's 600 problems are sourced from the test splits of Math-VR and Geo170K, together with an in-house geometry collection. We performed strict deduplication (text near-duplicate matching and image hashing) between the training data and GeoVAD-Bench to prevent train–test overlap.

We have incorporated the key experiments suggested by the reviewers into the revised PDF (pages 9–10), with all additions highlighted in blue.

Dear Reviewer enZ4,

Thank you again for your time and thoughtful feedback. We would like to kindly follow up on our response to your comments. We hope that the explanations and additional results have helped address your concerns, and we would be happy to provide any further clarification.

If your schedule permits, we would greatly appreciate any further feedback before the discussion period closes.

Best regards, The Authors

AuthorsOpenReview ↗

Kind Follow-up on Our Rebuttal Responses

Dear Reviewer enZ4,

We sincerely thank you for your thoughtful review, constructive suggestions, and the time dedicated to evaluating our work.

We have carefully addressed all your concerns with detailed responses, along with additional experiments and analyses in the revised manuscript. As the discussion period is nearing its end, we would greatly appreciate it if you could review our revisions when your schedule permits and let us know if any concerns remain or require further clarification.

Thank you again for your valuable feedback and contribution to the review process. We wish you all the best in your research.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Updates

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their time, thoughtful evaluations, and constructive feedback. We are encouraged by the reviewers’ consensus on the significance of formalizing the autonomy gap (GautonomyG_{\mathrm{autonomy}}), the utility of our multi-stage diagnostic framework (GeoVAD-Bench), and the principled alignment of our progressive SFT and Interleave-RL design with the diagnosed bottlenecks. During the rebuttal, we conducted new controlled experiments, human audits, and cross-judge evaluations to thoroughly address all raised concerns, as summarized below:

1. Direct evidence of closing the autonomy gap (enZ4–W1/Q1; 3Gub–W1/Q1). We added full No-Aux and GT-Aux evaluations across training stages. Relative to the base model (SenseNova-U1-8B), GeoWeave-8B reduces GautonomyG_{\mathrm{autonomy}} from 13.5 to 3.7 pp and turns VrealizedV_{\mathrm{realized}} from −6.5 to +4.9 pp. This confirms that our training not only strengthens general geometric reasoning (No-Aux: 43.8 → 57.7), but specifically enables the model to autonomously construct and leverage visual aids for net positive gains.

2. Causal attribution to visual assistance (enZ4–W2/Q2; uLtp–Q2). Against text-only controls trained on the exact same source data (with image segments removed), interleaved training yields +1.8 pp after SFT and +4.5 pp after RL. Counterfactual interventions on the first generated auxiliary diagram further drop accuracy by 7.8 pp (replaced with original), 7.0 pp (replaced with blank), and 6.6 pp (construction instruction masked), establishing that both the generated diagram and its instruction are causally essential.

3. Rigorous human validation and cross-judge consistency (enZ4–W3/Q4; uLtp–W1/Q1). A human audit of 200 trajectories confirms strong human–judge alignment across all diagnostic levels (86.60% for L1 perception, 80.28% for L2 diagram quality, 84.29% for L3 utilization, 93.26% for L4 reasoning, and 95.77% for L5 answer). A separate audit of 800+ reasoning steps yields 97.17% exact agreement (κ=0.8347\kappa = 0.8347, AC1=0.9659\mathrm{AC1} = 0.9659) for our SCA process verifier. Furthermore, re-evaluation with an independent non-Gemini judge (GPT-5.6-Terra) preserves all key conclusions: GeoWeave-8B ranks best overall, and SCA yields +16.0 pp in L4 and +4.8 pp in Process Avg. Together with increased explicit assertions per trajectory (20.26 → 28.28), these checks rule out judge bias or reward hacking via claim suppression.

We also provided detailed clarifications on L3's diagnostic scope, benchmark provenance and strict deduplication, failure-mode taxonomy (Others), and our first-error attribution rule, while outlining cross-backbone validation (e.g., Bagel) and local/open-source verifiers as immediate next steps.

All new controlled evaluations, counterfactual analyses, and human-validation results have been incorporated into the revised PDF (pages 9–10, highlighted in blue). We thank the reviewers again for their valuable insights, which have substantially strengthened the empirical rigor of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.