- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies Visual Chain-of-Thought for geometry problem solving, distinguishing the potential value of high-quality auxiliary diagrams, a model's ability to construct them autonomously, and their use in subsequent reasoning. The authors introduce GeoVAD-Bench, which contains 600 problems, and define an autonomy gap through three evaluation conditions: No-Aux, Auto-Aux, and GT-Aux. Model trajectories are evaluated along five dimensions: original-diagram perception, auxiliary-diagram quality, auxiliary-diagram utilization, reasoning correctness, and final-answer correctness.
Building on this diagnosis, the authors perform progressive supervised fine-tuning of SenseNova-U1-8B using geometric perception, diagram editing, and interleaved visual-textual reasoning data, followed by Interleave-RL with step-level process feedback. The resulting model, GeoWeave-8B, improves answer accuracy on GeoVAD-Bench from 37.3% to 62.6% and Process Avg. from 52.5% to 82.9%. The paper also reports results on several public benchmarks.
Critical strengths
- Distinguishing whether auxiliary diagrams are useful from whether models can autonomously realize their benefits is a valuable research direction. In Table 1, GT-Aux improves accuracy over No-Aux by 3.0–7.0 percentage points across the three systems, whereas Auto-Aux underperforms No-Aux for the two smaller models. This comparison shows that evaluating autonomous drawing systems solely by final-answer accuracy can obscure the potential value of valid auxiliary constructions.
- The process evaluation covers multiple stages before and after diagram construction, beyond final-answer correctness or generated-image quality. In particular, separately evaluating auxiliary-diagram quality and utilization helps distinguish incorrect construction from ineffective use of a constructed diagram. Table 2 further shows that the gap between correct and incorrect solutions is larger for reasoning-process scores than for diagram-quality scores, supporting attention to both construction and subsequent reasoning.
- The training ablations on the same base model provide positive evidence for the overall approach. In Table 5, SFT raises answer accuracy to 51.6%, reinforcement learning without Step-level Credit Assignment (SCA) further improves it to 60.3%, and the full method reaches 62.6%. SCA also improves reasoning-process accuracy from 46.3% to 66.3%. These results support the value of process feedback in this training setting.
Critical weaknesses
- The central claim of narrowing the autonomy gap lacks a direct evaluation consistent with its definition. Equation (2) defines this gap as the difference between GT-Aux and Auto-Aux accuracy for the same model, but Tables 3 and 5 do not report GT-Aux results for the trained GeoWeave model, so its post-training gap cannot be calculated. The absence of GeoWeave's No-Aux results also prevents assessing whether training increases the net benefit of autonomous visual operations relative to reasoning without generating auxiliary diagrams. The reported answer and process metrics support improvements in the overall system, but do not yet distinguish improvements in general geometric reasoning from increased benefits of autonomous visual operations. I suggest reporting the corresponding conditions for the final model or qualifying the claims about narrowing the gap to match the scope directly supported by the experiments.
- The error-attribution protocol for Figure 2 is insufficiently specified, limiting interpretation of the diagnostic distribution. Section 3.2 uses this distribution to motivate the four main process dimensions, but the main text and Appendix A.4 do not specify how an attribution category is selected when multiple error types occur in the same trajectory, or what cases fall under Others. It is therefore unclear whether the proportions reflect the first error, the dominant error, or another attribution rule. The four categories cover approximately 90% of cases, making them reasonable primary dimensions for analysis. Providing the attribution rule for trajectories with multiple errors and a few examples of Others would clarify the meaning of these diagnostic findings.
Questions for authors
- Could you report No-Aux and GT-Aux results for the final GeoWeave model and compare them with the corresponding base-model conditions in Table 1? If a full evaluation is costly, a comparison on the same fixed subset of problems with comparable prompts and budgets would also help assess the post-training realized value and autonomy gap.
- To clarify the methodology behind Figure 2, could you briefly explain what cases are included in Others and which attribution rule is used when multiple error types occur in the same trajectory?
3 comments
Thank you for the additional controlled evaluations and the clarification of the error-attribution protocol. The new results address my question about the post-training autonomy gap, and the explanation of first-error attribution and Others resolves my second clarification request. I will hold on to the current rating.
Thank you for your follow-up. We are glad that our additional evaluations and clarifications have addressed your questions. We appreciate your time and thoughtful feedback.
Response to Reviewer 3Gub
We thank the reviewer for recognizing the distinction between potential and realized visual-aid value, our multi-stage process diagnosis, and the training ablations. We address your questions below.
W1 / Q1. Direct measurement of the post-training autonomy gap
The controlled-evaluation results across the base model, SFT checkpoint, and final model are:
GeoWeave-8B narrows Gautonomy from 13.5 to 3.7 pp and turns Vrealized from −6.5 to +4.9 pp. Beyond strengthening general geometric reasoning, training makes autonomous visual assistance beneficial over the same model's No-Aux baseline. The SFT checkpoint shows the same trend.
W2 / Q2. Attribution rule for multiple errors and the meaning of Others
Thank you for noting that Figure 2's attribution protocol needs clarification. We use first-error attribution, and Others primarily covers three failure modes:
Representative examples: https://anonymous.4open.science/r/GeoWeave-05CC/others_showcase.pdf.
First-error attribution. Each failed trajectory is assigned to its first identifiable error or failure, avoiding ambiguity from downstream errors that stem from an earlier mistake. We will clarify both this rule and the definition of Others in the revised manuscript.
We have incorporated the key experiments suggested by the reviewers into the revised PDF (pages 9–10), with all additions highlighted in blue.