acceptodds
Withdrawn from ICLR 2027

Keep the Image in Mind: A Visual Board for Multimodal Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual reasoning agents must retain useful observations across steps without treating earlier interpretations as substitutes for direct visual evidence. Early visual misinterpretations can persist as faulty premises, leading to incorrect answers despite otherwise coherent reasoning. We introduce a visual reasoning harness centered on the Visual Board, a revisable textual workspace that maintains task-relevant observations and intermediate reasoning across steps. With continued access to the original image, the VLM uses the board to guide targeted visual inspection and revise earlier interpretations as new evidence becomes available. Deterministic history compression bounds textual context, while a comparably sized LLM critic reviews the VLM's interaction trace between rounds and provides feedback on reasoning validity. We evaluate the harness on Euclid 30K, SolidGeo, and the Chart QA subset, covering geometric, spatial, and chart reasoning. Our harness improves accuracy over the corresponding model-only baselines across all evaluated configurations, with gains exceeding 18 percentage points on the Chart QA subset. These results support the effectiveness of our Visual Board-centered harness for improving visual reasoning accuracy. Code is available at https://anonymous.4open.science/r/KeepInMind-1E60/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, Harness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TbskOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve the problem that early visual misinterpretations may mislead the reasoning and get wrong answers. The authors proposed a Visual Board centered harness for improving visual reasoning accuracy. The harness makes task-relevant visual premises, uncertain details, and intermediate reasoning explicit without requiring an exhaustive image description. Empirical experiments show that solver training improves both model-only and harness-based inference.

Critical strengths
  1. The paper addresses an important problem in visual reasoning, where incorrect visual observations can persist and mislead the following reasoning process. The proposed Visual Board provides a structured and revisable workspace for maintaining visual facts, uncertainties, and intermediate reasoning, while retaining access to the original image.

  2. The proposed harness improves performance over model-only baselines across multiple model configurations and benchmarks.

Critical weaknesses
  1. The proposed framework primarily combines existing techniques, including structured textual memory, visual inspection tools, history compression, and iterative solver–critic interaction, and the incremental contribution is unclear. Moreover, the main component ablations are conducted on only 50 Euclid 30K examples with a 32B solver, limiting the evidence that the Visual Board is consistently responsible for the improvements across different benchmarks and model scales.

  2. The comparison is unfair. The model-only baselines do not have access to the additional visual tools, critic feedback, and multiple reasoning rounds used by the proposed harness. As stated in Appendix A.1, the 4B configuration relies on Qwen3-VL-Flash for visual-tool calls and Qwen3-Max-Preview as the critic. So, the reason of reported gains are unclear that whether they attribute specifically to the proposed framework or the additional model capabilities and inference budget. Although the comparison with ProReason is useful, it is limited to one benchmark and does not establish a computation-matched comparison.

Questions for authors
  1. Do the benefits of the Visual Board remain consistent across different benchmarks and model scales? The current component ablations are limited to only 50 examples with a 32B solver.

  2. How often does the Visual Board actually help correct previously mistaken visual observations? A quantitative analysis of successful and unsuccessful premise revisions would help to understand why it works.

  3. Why there are hashing-like strings in this paper (Line 790, Line 830)? Could the authors clarify this?

  4. The released cold_start/review_euclid30k_8b.py records Codex-based answer-equivalence judgments that revise correctness labels and recompute accuracy, while Section 6 states that AI tools were used only for language polishing and code debugging. Were these judgments used in the reported Euclid30K results, and if so, how did they affect accuracy and how does this align with the AI Use Statement?

5 comments

We thank the reviewer for the constructive feedback. We address the concerns with expanded 500-question component comparisons across three configurations, Board-format comparisons on Euclid30K and ChartQA, including ChartQA evaluations before and after SFT, an All-4B system, Board-update analysis, and evaluation clarification.

1. W1 & Q1: Core Contribution and Expanded Ablations

Our key contribution is the Visual Board, a structured linguistic representation of image content, rather than a general-purpose store of dialogue history. It separates image-grounded observations and unresolved visual uncertainties from downstream reasoning, maintaining a compact representation through updates rather than unrestricted accumulation of reasoning traces. Although its persistent, shared implementation may resemble memory, the central design choice is how visual evidence is represented and maintained to support subsequent reasoning, not persistence or sharing alone.

To evaluate the value of this design, we expanded the original 50-example pilot to 500 Euclid30K questions with All-4B(Qwen3VL-4B for solver, Qwen3-4B for critic), All-8B(Qwen3-VL-8B for solver, Qwen3-8B for critic), and Qwen3.8-Flash(Qwen3.8-VL-Flash for solver and critic) configurations. Within each model group, the shared free-form scratchpad (board_free), no-Board retry baseline (no_board), and default Board retain the same Solver, visual-tool backend, and Critic.

In board_free, the model writes its own free-form textual notes as shared memory, without the Visual Board's prescribed structure.

The comparisons use the same maximum of 3 rounds, tool-call limits, stopping criteria, and maximum output-token setting. Actual reasoning-token consumption varies, so we report resource use rather than describe the runs as strictly total-token-budget matched.

ConfigurationSettingAccuracy (%)Tokens/questionTime/question (s)
All-4BNo Board / retry30.0060,636.1364.73
All-4BShared free-form notes35.9454,601.0064.61
All-4BDefault Board49.0055,825.1956.87
All-8BNo Board / retry51.4066,722.68153.56
All-8BShared free-form notes49.4048,313.90112.33
All-8BDefault Board58.7550,421.00103.16
Qwen3.8-FlashNo Board / retry60.0069,240.2285.56
Qwen3.8-FlashShared free-form notes61.8054,604.4964.36
Qwen3.8-FlashDefault Board68.4742,775.7561.66

Accuracies use the retained scoring sets, which can be smaller than the sampled count. The All-4B default-versus-free-form comparison uses 498 common questions, with run failures counted as incorrect. Resource figures are original run aggregates.

Structured Board vs. shared notes.

ConfigurationDefault minus free-form accuracy (pp)95% paired bootstrap CI (pp)Comparison scope
All-4B+13.05[+8.43, +17.67]244/498 vs. 179/498 correct
All-8B+9.35Not reportedAggregate scoring sets
Qwen3.8-Flash+6.67Not reportedAggregate scoring sets

All-4B uses 498 common questions, counts run failures as incorrect, and uses 100,000 paired bootstrap resamples. There are 105 default-only correct and 40 free-form-only correct questions. Its difference is calculated from unrounded counts. pp denotes percentage points.

The results favor structured visual evidence over free-form shared notes in these Euclid configurations, with paired statistical support for All-4B.

Cross-benchmark comparisons after SFT. We additionally compare inference settings for the same Qwen3-VL-4B Solver after 200 SFT steps on 1,556 trajectories from the Euclid train split. Euclid testing uses the separate evaluate split; ChartQA uses the 826-question bar-chart subset, with no ChartQA examples used for SFT. Both Solver-only tool-use settings below retain a Board and omit the Critic; they differ in whether the Board uses free-form notes or the prescribed structure.

Euclid30K inference settingBoardCriticSFT 4B accuracy (%)
Direct QANoneNo42.80
Solver-only + ToolsFree-formNo55.20
Solver-only + ToolsDefaultNo55.80
Full harness, heterogeneousDefaultYes57.03

ChartQA before and after SFT: separating training gains from Board-format differences. We additionally evaluate the base Solver under the same ChartQA inference settings. Within each row below, tool access, visual-tool backend, Board format, and Critic setting are held fixed between the base and SFT Solvers.

ChartQA Inference SettingBoardCriticBase 4B (%)SFT 4B (%)SFT Gain (pp)
Direct QA (no tools)NoneNo57.7570.58+12.83
Solver-only + ToolsFree-formNo77.2483.78+6.54
Solver-only + ToolsDefaultNo78.4582.81+4.36
Full harness (heterogeneous)DefaultYes78.6984.87+6.18

Base means no additional SFT; all SFT rows use the same 200-step checkpoint.

Training gains within a fixed Board format. Read each ChartQA row from Base to SFT: accuracy improves under both Board formats without Critic feedback, as well as under direct QA and full-harness inference. This supports geometry-to-chart transfer across inference settings.

In the free-form setting, the Solver itself decides what information to store and how to organize it, without the Board's prescribed structure. Our interpretation is that training on harness rollouts may help the Solver internalize Board-related knowledge about selecting, organizing, and using visual evidence, which it can then apply through self-organized notes. The improvement over the base Solver under the same free-form setting is consistent with this interpretation and with transfer beyond geometry.

2. W2: Helper-Model Capacity and Inference Resources

We distinguish stronger helper models from additional inference. In the original heterogeneous system, "4B" denotes the Solver: Qwen3.0-VL-Flash serves the visual tools and Qwen3-Max-Preview serves as Critic.

To address helper-model capacity, we added an All-4B configuration using Qwen3-VL-4B-Instruct for both the Solver and visual tools, and Qwen3-4B for the Critic:

ConfigurationEuclid30K (%)SolidGeo (%)ChartQA (%)
Standalone Qwen3-VL-4B-Instruct39.4032.1157.75
All-4B harness49.0040.8477.97
Improvement (percentage points)+9.60+8.73+20.22

Substantial gains therefore persist without larger helper models. This is a matched-size system comparison, not an equal-compute comparison.

Resource trade-offs within each configuration. The table compares default Board against each reference using the accuracy and token statistics in Section 1.

ConfigurationReferenceAccuracy difference (pp)Token change (%)
All-4BNo Board / retry+19.00-7.9
All-4BFree-form Board+13.05+2.2
All-8BNo Board / retry+7.35-24.4
All-8BFree-form Board+9.35+4.4
Qwen3.8-FlashNo Board / retry+8.47-38.2
Qwen3.8-FlashFree-form Board+6.67-21.7

The Board improves observed accuracy with fewer tokens than retry in all three configurations. Relative to free-form notes, it uses slightly more tokens for All-4B and All-8B and fewer for Qwen3.8-Flash.

Reviewer_K3iBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Visual Board-centered framework for multimodal reasoning that combines revisable textual memory, targeted visual inspection, history compression, and a text-only LLM critic. The framework improves accuracy on geometry and chart reasoning benchmarks, with additional gains from supervised fine-tuning.

Critical strengths
  1. The paper points out an important issue, which is incorrect visual observations can persist as faulty premises throughout multi-step reasoning and uses a harness-based method to solve.

  2. The framework consistently improves over the corresponding model-only baselines across the evaluated model configurations and benchmarks.

Critical weaknesses
  1. The baselines are limited. Most comparisons are against Qwen models performing direct QA. It would be helpful to compare against Qwen with the same tool access and inference budget but without the proposed harness, as well as Qwen trained on the same source data using conventional tool-use SFT. These controls would help isolate the contributions of the harness and harness-aware training.

  2. Ablation study is also limited. Existing ablations cover only 50 examples, and controlled training comparisons are missing. Conventional reasoning or tool-use SFT with matched source data, teachers, and training budgets would help distinguish the benefits of harness-aware training from general supervision and distillation gains. And the module of LLM should also take into consideration, which could prove the contribution of this part.

  3. The module of LLM is confusing as the critic only accesses text, it cannot independently verify visual premises and may accept incorrect but internally consistent descriptions.

Questions for authors

see the above weakness

3 comments

We thank Reviewer K3iB for the constructive feedback. We address the concerns with 500-question component ablations, completed base-versus-SFT tool-use comparisons on Euclid30K and ChartQA, and published VTool-R1 baselines. We also clarify the Critic's role as a reasoning reviewer, rather than a visual verifier.

1. W1 & W2: Tool-Use Baselines and Expanded Ablations

Our contribution is the structured representation and revision of visual premises, beyond tool access or feedback alone. We expanded the original 50-question pilot to 500 Euclid30K evaluation questions across three configurations. Within each configuration, the following settings use the same Solver and available tools:

The table identifies Board format and Critic access for each setting: no_board is retry without a Board; board_free uses shared free-form notes; w/o_critic removes Critic feedback from the full Board setting.

The Board comparisons retain the same Critic, maximum of 3 rounds, tool-call limits, stopping criteria, and maximum output-token setting. Reasoning-token consumption is not independently capped, so we report actual tokens rather than claim identical total inference budgets.

Base ModelSettingBoardCriticAcc. (%)Tokens/q
Qwen3-VL-4B (All-4B)w/o_criticFullNo36.8044,274.37
no_boardNoneYes30.0060,636.13
board_freeFree-formYes35.9454,601.00
defaultFullYes49.0055,825.19
Qwen3-VL-8B (All-8B)w/o_criticFullNo50.8028,374.05
no_boardNoneYes51.4066,722.68
board_freeFree-formYes49.4048,313.90
defaultFullYes58.7550,421.00
Qwen3.8-Flashw/o_criticFullNo69.8017,187.12
no_boardNoneYes60.0069,240.22
board_freeFree-formYes61.8054,604.49
defaultFullYes68.4742,775.75

Tokens/q is mean total token consumption per evaluated question. Aggregate scoring denominators can differ. All-4B default/board-free accuracies use 498 common questions.

Structured Board vs. shared notes.

ConfigurationDefault minus free-form accuracy (pp)95% paired bootstrap CI (pp)Comparison scope
All-4B+13.05[+8.43, +17.67]244/498 vs. 179/498 correct
All-8B+9.35Not reportedAggregate scoring sets
Qwen3.8-Flash+6.67Not reportedAggregate scoring sets

All-4B uses 498 common questions, counts run failures as incorrect, and uses 100,000 paired bootstrap resamples. There are 105 default-only correct and 40 free-form-only correct questions. Its difference is calculated from unrounded counts. Other differences are descriptive aggregate comparisons. pp denotes percentage points.

The results favor structured visual evidence over free-form shared notes in these Euclid configurations, with paired statistical support for All-4B. The full Board also outperforms no-Board retry with fewer tokens in all three configurations.

Gains without larger helpers. All-4B uses Qwen3-VL-4B-Instruct as Solver and visual-tool backend and Qwen3-4B as Critic.

BenchmarkStandalone 4B accuracy (%)All-4B harness accuracy (%)Harness minus standalone (pp)
Euclid30K39.4049.00+9.60
SolidGeo32.1140.84+8.73
ChartQA57.7577.97+20.22

This matched-size system comparison demonstrates that the performance gains persist without relying on larger auxiliary models.

2. W1 & W2: SFT Gains and Tool-Use Comparisons

We compare the base Qwen3-VL-4B Solver with the checkpoint after 200 SFT steps on 1,556 geometry trajectories from the Euclid training split, using the separate held-out split for Euclid evaluation. No ChartQA examples are used for SFT. The completed evaluations distinguish direct QA, Solver-only tool use with either Board format, and the full harness. Direct QA uses no tools, Board, or Critic.

Euclid30K Inference SettingBoardCriticBase 4B (%)SFT 4B (%)SFT Gain (pp)
Direct QANoneNo39.4042.80+3.40
Solver-only + ToolsFree-formNo31.2055.20+24.00
Solver-only + ToolsDefaultNo36.8055.80+19.00
Full harness (heterogeneous)DefaultYes52.6157.03+4.42

Euclid30K: SFT Enables the Solver to Internalize Board Construction and Maintenance. The Free-form Board allows the Solver to autonomously decide what information to record and how to maintain its notes, resembling a memory-augmented tool-use setting. After SFT, the Free-form Board achieves a 24.00 pp improvement, while the default Board gains 19.00 pp, both without Critic feedback. These results suggest that SFT enables the Solver to learn effective Board construction and maintenance behaviors. Notably, under the Solver-only + tools configuration, the default Board outperforms the Free-form Board by only 0.6 pp, indicating that the fine-tuned model can independently maintain useful visual reasoning notes with minimal reliance on predefined Board structure.

ChartQA: the gains transfer beyond geometry. On the 826-question Chart Split, the base and SFT Solvers use the same tool access, visual-tool backend, Board format, and Critic setting within each row.

ChartQA Inference SettingBoardCriticBase 4B (%)SFT 4B (%)SFT Gain (pp)
Direct QA (no tools)NoneNo57.7570.58+12.83
Solver-only + ToolsFree-formNo77.2483.78+6.54
Solver-only + ToolsDefaultNo78.4582.81+4.36
Full harness (heterogeneous)DefaultYes78.6984.87+6.18

In both tables, Base means no additional SFT, and all SFT rows use the same 200-step checkpoint. Full-harness rows use the original heterogeneous helpers, not the All-4B configuration in Section 1.

On ChartQA, SFT adds 6.54 pp with free-form Board and 4.36 pp with default Board, again without Critic feedback. Together with the 12.83 pp direct-QA gain, these results support geometry-to-chart transfer across inference settings. Inference-time support remains useful after SFT: the full harness exceeds direct QA at the same checkpoint by 14.23 pp on Euclid30K and 14.29 pp on ChartQA.

External tool-use references. VTool-R1 [1], Table 1 reports the following results on the same Chart Split:

Published BackbonePure Run (%)Prompted Tool Use (%)VTool-R1, RL-Trained (%)
Qwen2.5-VL-3B51.824.664.0
Qwen2.5-VL-7B76.253.480.7
Qwen2.5-VL-32B88.085.086.7

The first two columns use models without additional RL training; the last reports RL-trained checkpoints.

Our SFT 4B Solver with free-form/default Board and no Critic reaches 83.78%/82.81%, numerically above VTool-R1-7B's 80.7% by 3.08/2.11 pp. The full heterogeneous system reaches 84.87%, compared with VTool-R1-32B's 86.7%.

Training control. Limited compute during rebuttal prevented a separate conventional tool-use SFT run matched on source questions, teachers, and training budgets. We prioritized the completed base-versus-SFT evaluations above. They support the benefits of training on harness trajectories.

3. W2 & W3: The Critic Reviews Reasoning, Not Images

The Board is a structured linguistic representation of image content. It separates observations and uncertainties from derived reasoning, enabling an LLM to check whether the reasoning follows from the recorded premises. As stated in the Method section, the Critic neither generates an independent solution nor modifies the Board. It flags unsupported assumptions or derivations and provides feedback; visual reinspection, revision, and answer generation remain the VLM Solver's responsibility.

We agree that a visually incorrect but internally consistent description can pass textual review. Critic acceptance is not independent verification of the image or answer. Its contribution is complementary reasoning feedback.

BenchmarkSolver configurationNo Critic accuracy (%)With Critic accuracy (%)With minus without (pp)
Euclid30KBase All-4B36.8049.00+12.20
Euclid30KBase All-8B50.8058.75+7.95
Euclid30KQwen3.8-Flash69.8068.47-1.33
Euclid30KSFT 4B55.8057.03+1.23
ChartQABase 4B78.4578.69+0.24
ChartQASFT 4B82.8184.87+2.06

All rows retain the default Board. Positive differences favor Critic feedback.

These results are consistent with Section 2: SFT improves Solver-only tool use, while Critic feedback can provide additional gains. However, the benefit of Critic feedback varies across model configurations and tasks. Qwen3.8-Flash achieves higher accuracy with fewer tokens without the Critic, suggesting that its stronger Solver can perform reliable reasoning independently. In contrast, Qwen3 models still benefit from Critic feedback, which provides an additional layer of reasoning verification and helps improve robustness.

Thank you again for helping us strengthen the component analysis and the evidence for complementary training and inference-time gains.

[1] Wu M, Yang J, Jiang J, et al. VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use. ICLR 2026. Table 1.

Reviewer_Ph34OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a real problem in visual agents: a wrong image description can persist through later reasoning. Its Visual Board separates image facts, uncertain facts, and reasoning in a shared, revisable state. A Solver uses visual tools and receives feedback from a text-only Critic; history compression limits context growth. The paper also fine-tunes a model on successful teacher traces. Results cover Euclid, SolidGeo, and an 826-question bar-chart subset of ChartQA.

Critical strengths
  1. The problem matters. A visual agent can keep reasoning from a false observation. Recording uncertain facts and revisiting the image is a sensible response, and the failure traces help explain when it works.
  2. The system delivers clear gains. Accuracy improves across the tested model configurations. The matched 8B ProReason comparison, Board/tool/compression ablations, and model-only SFT control provide useful evidence.
Critical weaknesses
  1. The Board's structure is not separated from the benefit of shared memory. ProReason already combines active perception, memory, and feedback. The key question is whether separating visual facts, uncertainties, and reasoning adds value. The 50-question ablation compares shared Board, solver-local Board, and no Board, but not shared free-form notes. Sharing raises accuracy from 60% to 64%, a net gain of two correct answers; this small pilot leaves the effect uncertain.
  2. The small-model comparison does not establish system efficiency. The 4B Solver uses Flash visual tools and a Max-Preview Critic. Its accuracy therefore reflects the whole system. The paper reports pilot timing results, but a matched comparison counting all helper-model calls, tokens, latency, and cost is needed to establish an efficiency advantage.
Questions for authors
  1. Does the Board beat an ordinary shared scratchpad and a retry baseline with the same Solver, tools, Critic, and total token budget? Please test the key comparison on more than the 50-question pilot and report uncertainty on the paired difference.

  2. What are the total model calls, tokens, latency, and cost per solved question? Please include all helper models and separate the heterogeneous 4B/Flash systems from the same-size configurations. The existing pilot timing results do not answer this comparison.

  3. Which SFT test sets have verified source separation from training? Appendix C.2 acknowledges possible SolidGeo overlap; please separate that result from the cleaner ChartQA bar-chart transfer result.

4 comments

We sincerely thank Reviewer Ph34 for the constructive feedback and for recognizing the importance of revisiting uncertain visual observations and the empirical gains of our system. We address your concerns with expanded Board ablations, a matched-size All-4B comparison, and an explicit analysis of SFT data overlap.

1. W1 & Q1: Visual Board vs. Shared Scratchpad and Retry Baselines

The Board is designed as a structured linguistic representation of image content, and the following comparisons test whether shared free-form notes can substitute for it.

Following your suggestion, we conducted expanded ablations on 500 questions from the Euclid30K evaluation set, including a shared free-form scratchpad (board_free) and a no-Board retry baseline (no_board). The aggregate table below reports accuracy over the items retained for scoring in each setting, so the scoring denominator may be smaller than the sampled count. For the 4B default Board vs. board-free comparison, both accuracies and the paired analysis use the same 498 common questions, with run failures counted as incorrect. Within each model group, the two baselines and the full Board retain the same Solver, visual-tool backend, and Critic configurations. We additionally evaluate a concise Board and removal of the Critic.

Evaluation controls. The matched comparisons use the same maximum of 3 rounds, tool-call limits, stopping criteria, and maximum output-token setting. However, we cannot enforce a separate hard cap on the models' reasoning-token consumption, so these controls do not guarantee an identical total token budget. We therefore report actual total token usage and evaluate the accuracy-token trade-off rather than describing the runs as strictly total-token-budget matched.

S = Solver calls; C = Critic calls; V = visual-tool backend calls. API call counts are run totals, whereas tool calls, tokens, and elapsed time are averages per evaluated question. Tool invocations and backend model API calls are reported separately. Tokens/q denotes token consumption, not monetary cost.

Base ModelSettingBoardCriticAcc. (%)Tool calls/qTotal API calls (S / C / V)Tokens/qTime/q (s)
Qwen3-VL-4B (All-4B)no_boardNoneYes30.0011.315799 / 4293 / 81860,636.1364.73
board_freeFree-formYes35.949.735202 / 3173 / 86554,601.0064.61
w/o_criticFullNo36.806.044028 / 0 / 183844,274.3743.79
board_conciseConciseYes40.4010.625655 / 3609 / 146277,257.1883.46
defaultFullYes49.009.894462 / 1696 / 404455,825.1956.87
Qwen3-VL-8B (All-8B)no_boardNoneYes51.409.974978 / 3555 / 179566,722.68153.56
board_freeFree-formYes49.407.823752 / 1649 / 145348,313.90112.33
w/o_criticFullNo50.802.442366 / 0 / 39128,374.0577.36
board_conciseConciseYes58.207.834425 / 3616 / 119668,391.39160.86
defaultFullYes58.754.344298 / 1575 / 90550,421.00103.16
Qwen3.8-Flashno_boardNoneYes60.0011.203756 / 2475 / 419769,240.2285.56
board_freeFree-formYes61.808.732807 / 1990 / 196954,604.4964.36
w/o_criticFullNo69.802.591498 / 0 / 90717,187.1226.15
board_conciseConciseYes67.209.142990 / 1987 / 387162,733.7068.50
defaultFullYes68.477.682784 / 2615 / 181342,775.7561.66

Paired uncertainty: default Board vs. board-free. For 4B, we performed 100,000 paired bootstrap resamples using the reconciled, reviewed question-level results on 498 common questions. These accuracy estimates agree with the corresponding entries in the main table.

ModelPaired questionsdefault Board correct / N (Acc.)board-free correct / N (Acc.)Accuracy difference (pp)95% paired bootstrap CI (pp)
4B498244/498 (49.00%)179/498 (35.94%)+13.05[+8.43, +17.67]
Paired outcomeQuestions
Both correct139
Default Board only correct105
Free-form Board only correct40
Both incorrect214
Total common questions498

Two questions present in the free-form run are excluded; run failures count as incorrect. The difference uses unrounded counts: 100 x (105 - 40) / 498 = 13.05 pp. The interval lies entirely above zero, providing statistical support for a positive default Board vs. board-free accuracy difference under this evaluation protocol.

Key takeaways: The structured Board outperforms shared free-form notes in these Euclid configurations, with paired statistical support for All-4B. It also outperforms no-Board retry with fewer tokens and lower elapsed time; Section 2 further tabulates the trade-offs, critic removal hurts All-4B and All-8B but helps Qwen3.8-Flash.

In response to the comparison with ProReason, our claim concerns this visual representation rather than persistence or sharing alone. The board_free ablation and paired analysis support its value over ordinary shared free-form notes under the evaluated configuration.

Design insight: a structured linguistic space for visual evidence. The Board is designed to represent image content in language, rather than to serve as a general-purpose store of dialogue history. It separates image-grounded observations and unresolved visual uncertainties from downstream reasoning, maintaining a capacity-bounded representation through updates rather than unrestricted accumulation of reasoning traces. Although its persistent, shared implementation may resemble memory, the central design choice is how visual evidence is represented and maintained for subsequent reasoning.

2. W2 & Q2: Matched-Size 4B Comparison and System Resource Use

To directly address the concern that the 4B Solver benefits from larger helper models, we added an All-4B configuration: Qwen3-VL-4B-Instruct serves as both Solver and visual-tool backend, and Qwen3-4B serves as Critic.

ConfigurationSolverVisual-tool backendCriticEuclid30KSolidGeoChartQA
Standalone 4BQwen3-VL-4B-InstructNoneNone39.4032.1157.75
Original heterogeneous systemQwen3-VL-4B-InstructQwen3.0-VL-FlashQwen3-Max-Preview52.6137.0878.69
All-4B systemQwen3-VL-4B-InstructQwen3-VL-4B-InstructQwen3-4B49.0040.8477.97

The benchmark columns above report accuracy (%). The differences below are All-4B minus the named reference; positive values favor All-4B.

Reference configurationEuclid30K difference (pp)SolidGeo difference (pp)ChartQA difference (pp)
Standalone 4B+9.60+8.73+20.22
Original heterogeneous system-3.61+3.76-0.72

Substantial gains over standalone inference persist without larger helper models. This is a matched-size comparison, not an equal-compute comparison.

Efficiency as an accuracy-token trade-off. Section 1 reports all three model-call roles, tokens, and elapsed time. The following table summarizes the default Board's differences from each reference within the same configuration.

ConfigurationReferenceAccuracy difference (pp)Token change (%)
All-4BNo Board / retry+19.00-7.9
All-4BFree-form Board+13.05+2.2
All-8BNo Board / retry+7.35-24.4
All-8BFree-form Board+9.35+4.4
Qwen3.8-FlashNo Board / retry+8.47-38.2
Qwen3.8-FlashFree-form Board+6.67-21.7

Accuracy difference = default Board minus reference. Token change = 100 x (default tokens / reference tokens - 1); negative values mean fewer tokens. The All-4B free-form accuracy comparison uses 498 common questions; other accuracy differences use the reported aggregate scoring sets.

The Board consistently achieves higher accuracy with lower token consumption than the retry baseline across all three configurations. Compared with free-form notes, it incurs slightly higher token costs for All-4B and All-8B, while reducing token usage for Flash. Overall, our default method achieves a competitive trade-off between accuracy and token efficiency.

3. Q3: SFT Data Separation and SolidGeo Overlap

Euclid train/evaluation separation. All 1,556 SFT rollout samples were generated from the Euclid train split, whereas our Euclid tests use the evaluate split. The evaluation questions were not used to construct the SFT training trajectories.

As your suggestion, we audited the 1,556 Euclid SFT rollout samples against the evaluation data using a question-image-answer matching criterion: an evaluation item was classified as overlapping only when its question, image, and answer all matched the same training sample. Under this criterion, we identified 89 overlapping items among SolidGeo's 2,206 evaluation questions (4.03%). We distinguish the original results from conservative bounds accounting for this detected overlap:

SolidGeo evaluation / metricQuestions in denominatorAccuracy (%)Result type
Base 4B2,20632.11Reported full-set score
Harness without SFT2,20637.08Reported full-set score
Harness + SFT, unfiltered2,20657.89Reported full-set score
SFT worst-case penalty for detected overlap2,20653.86Conservative bound; +16.78 pp vs. harness without SFT
SFT overlap-excluded subset2,11756.12Lower bound, not measured subset accuracy

The worst-case penalty assigns zero credit to all 89 overlapping items while retaining the original denominator: 57.89 - 100 x 89/2,206 = approximately 53.86%, still 16.78 percentage points above Harness without SFT. Alternatively, assuming all overlapping items were answered correctly and removing them yields a 56.12% lower bound on the remaining 2,117 questions.

ChartQA transfer. No matches were found under the same question-image-answer matching criterion. ChartQA evaluates bar-chart reasoning rather than geometry and contains no SFT training examples.

ChartQA configurationAccuracy (%)SFT + harness minus this reference (pp)
Base 4B, direct QA57.75+27.12
Base 4B, harness78.69+6.18
SFT 4B, direct QA70.58+14.29
SFT 4B, harness84.87--

The comparison with base-model direct QA combines training and inference-system differences. Relative to SFT direct QA, the harness adds 14.29 pp using the same trained Solver. The same-harness comparison shows an observed SFT gain of 6.18 pp, supporting transfer beyond geometry.

Thank you again for helping us strengthen the comparison between the Board's structured visual representation and shared free-form notes, add matched-size evidence, and clarify the scope of our efficiency and SFT generalization claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.