acceptodds
Under review as a conference paper at ICLR 2027

Visual Reasoning with Chain-of-Coordinates

Abstract

We introduce Chain-of-Coordinates (CoC), a representation for visual reasoning in which vision-language models (VLMs) autoregressively generate sequences of image coordinates. We provide a theoretical analysis of CoC, covering its representability, learnability, geometric approximation error, and computational advantage over direct answering on a family of connectivity problems. We introduce a two-stage reinforcement learning (RL) framework with a Fréchet-based geometric reward. Across six tasks, CoC improves curve tracing and consistently outperforms direct-answer training on reasoning tasks, with further gains from RL. Additional analyses show that CoC is most beneficial when connectivity cannot be inferred from direct visual cues, while the effectiveness of CoC prompting depends on a model's underlying tracing ability. Together, these results establish CoC as a paradigm for connecting visual perception and reasoning through explicit coordinate sequences.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Reasoning, Chain-of-Thought, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_T4cQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Chain-of-Coordinates (CoC), a representation that enables vision-language models (VLMs) to express curves, paths, and intermediate spatial states through textual coordinate sequences. The paper provides theoretical analyses of representability, local-rule learnability, geometric approximation error, and the computational complexity of endpoint connectivity. It further introduces a two-stage SFT+GRPO training framework with a geometric reward based on continuous Fréchet distance. Experiments across six tasks involving drawing, writing, line connection, nonsemantic tracing, circuit connectivity, and curve containment demonstrate improvements in tracing quality and visual reasoning performance. However, these benefits depend on the availability of direct visual shortcuts and the model's underlying tracing capability.

Critical strengths

Geometrically meaningful intermediate representation. Coordinate sequences serve as both executable outputs and intermediate reasoning states. The continuous Fréchet reward accommodates different sampling densities, providing a more appropriate measure of trajectory similarity than strict point-wise correspondence.

Connection between theoretical analysis and empirical validation. The analyses of local-rule learnability, geometric error decomposition, and endpoint connectivity provide theoretical support beyond empirical intuition. Experiments across six tasks further demonstrate that CoC is particularly beneficial when direct visual shortcuts, such as distinctive colors or appearance cues, are unavailable.

Critical weaknesses

Insufficiently controlled attribution of performance gains. Compared with direct-answer SFT, CoC training benefits from additional trajectory annotations and longer output sequences, as well as an additional geometric reward during RL. The current comparisons do not adequately disentangle whether the improvements arise from the coordinate-based reasoning representation itself or from richer trajectory supervision and additional training signals.

Limited practical implications of the computational complexity analysis. Theorem 12 relies on an artificially constructed family of block-based curves, specific AC0/TC0/NC1 computational assumptions, and the conditional separation TC0 ≠ NC1. These theoretical results do not directly establish that CoC provides superior reasoning capabilities over direct answering in real-world VLMs.

Questions for authors

Reliability of the Fréchet-based geometric reward. The reward allows arbitrary starting points and traversal directions for closed curves, as well as Hungarian matching for multiple curves. Could these invariances assign high rewards to geometrically similar but topologically incorrect trajectories, particularly in cases involving multiple curves, crossings, or incorrect curve correspondences? Please provide an analysis of the correlation between geometric rewards and final-answer correctness or human-assessed trajectory correctness, along with sensitivity analyses under these challenging conditions.

4 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank the reviewer for the thoughtful review, and for recognizing CoC's role as executable output and reasoning state, and the connection between our theory and experiments. We address each point below.


W1. Attributing the performance gains

We agree that the representation should be separated from the extra supervision and training signal. The reviewer names three possible sources of the gain: trajectory annotations, longer outputs, and the geometric reward.

Trajectory annotations and output length. We trained an answer-then-trace model with exactly the same trace annotations and output length as CoC. It generates the answer before the trace, so the answer cannot depend on the trace. All models use the same images, base model and training budget (mean ± std over 3 training seeds):

MethodTrace labelsTrace before answerTunnel Vision (125)CurveBench (45)TRACER-Bench (50)
Direct-answer SFT✗✗74.9 ± 2.845.9 ± 3.482.7 ± 4.2
Answer-then-trace SFT✓✗73.9 ± 1.749.6 ± 5.185.3 ± 4.2
CoC SFT✓✓93.3 ± 1.277.0 ± 2.691.3 ± 1.2

Answer-then-trace is trained on the matching trace annotations and compute, and produces outputs of the same length as CoC, so any benefit of the extra supervision, including its use as an auxiliary task that improves the learned features, is available to both models. The only difference is the order: because generation is autoregressive, the answer cannot be computed from a trace produced after it. Comparing the two therefore separates the effect of the extra supervision from the effect of reasoning through the trace. Answer-then-trace performs close to direct answering on every task (p≥0.36p \geq 0.36, Welch's tt-test over seeds) and far below CoC on Tunnel Vision (−19.4-19.4 pt, p=1×10−4p = 1 \times 10^{-4}) and CurveBench (−27.4-27.4 pt, p=0.004p = 0.004). With matched data and compute, the trace annotations and extra tokens therefore bring little on their own; the gain comes from the additional representation power that CoC provides by computing the answer through the trace, which is empirical evidence for the practical relevance of Theorem 12.

Geometric reward. All results in the table are measured after SFT, before any RL. The improvement over direct answering therefore comes from the representation itself, not from the additional reward. RL with the geometric reward then refines the model: it improves trace fidelity (Fréchet distance on Quick, Draw!: 0.266 → 0.240) and adds further gains on Tunnel Vision, TRACER-Bench and BabyVision.

Supporting evidence. Two existing results lead to the same conclusion. First, on Tunnel Vision Circuit Connections the trace supervision is identical in the Single-Color and Unique-Color settings, yet CoC’s advantage drops from +18.4 pt (p = 6.6 × 10⁻⁵) to +2.4 pt (p = 0.45) once colors reveal the connections. The gain follows whether tracing is needed, not how much supervision is given; this also explains the smaller gap on TRACER-Bench, where paths often have distinct colors. Second, CoC helps even with no trace supervision at all: CoC prompting alone improves frontier models (e.g. GPT-5.6 Sol on CurveBench, 80.0 → 95.6; Table 2).


W2. Practical implications of Theorem 12

We agree that Theorem 12 is a conditional result on a constructed family. Its purpose is to explain the mechanism behind CoC's advantage, not to guarantee performance. Direct answering has to resolve global connectivity within a fixed number of parallel steps, whereas CoC carries spatial state forward one segment at a time. The separation builds on the S₅ construction of Barrington (1989) and Li et al. (2024). Our contribution is to show that coordinate traces suffice for this separation with AC⁰ computation per token.

The theory makes three predictions that our results on trained VLMs support:

  • The answer must be computed through the trace. If the gain came only from learning the trace, generating it after the answer would work as well. It does not: answer-then-trace performs like direct answering (W1).
  • CoC should help most when connectivity must be traced. The gain is +18.4 pt on Single-Color circuits but only +2.4 pt when unique colors make tracing unnecessary. The real benchmark closest to the Theorem 12 family, Tunnel Vision Circuit Connections, where wires permute through crossings, is also where CoC's gain over direct answering is largest.
  • The cost of CoC should grow with path length. Tokens per question range from 74 on Tunnel Vision to about 1,270 on BabyVision's long, dense paths, in line with the O(Nimglog⁡Nimg)O(N_{img}\log N_{img}) term in Theorem 12(ii).

We therefore present the theory as an explanation of why and when CoC helps, consistent with these results, rather than as a guarantee for every real-world setting.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Q1. Reliability of the Fréchet-based geometric reward

We address each failure mode the reviewer raises below, then report the correlation analyses requested. In short, the reward reliably penalizes wrong turns and wrong curves; the reviewer's concern holds for two specific cases, which we quantify.

1. Start-point and direction invariance. This cannot admit a topologically incorrect trace. The invariance applies only to closed curves, where every start point and direction traces exactly the same set. Open reasoning traces (Tunnel Vision, TRACER-Bench) are scored in a fixed direction.

2. Crossings. Wrong turns are heavily penalized. Continuous Fréchet distance is the maximum separation along the best traversal, not an average, so a trace that takes the wrong branch is penalized by the full distance between the branches. Measured on open traces: correct traces with small vertex noise (σ ≤ 20 grid units) keep a median rFr_F of 0.68–0.92, while a branch switch at a crossing drops it to 0.01–0.03. Correct and incorrect traces are separated with AUROC 0.992 on Tunnel Vision and 1.000 on TRACER-Bench.

Exception: near-parallel curves (as seen in Tunnel Vision). When a neighbouring wire runs 25–50 grid units away, 35% of jumps onto it keep rFr_F ≥ 0.5.

3. Incorrect curve correspondence. For queried curves this cannot occur. In Tunnel Vision and TRACER-Bench, each queried trace is compared with its own reference curve, with no Hungarian matching. Tracing the wrong curve is therefore penalized: a jump to the nearest neighbouring curve drops rFr_F to 0.01–0.15.

4. Multiple curves with Hungarian matching (CurveBench). Errors within the curves are penalized: merging two curves gives rFr_F 0.61, and swapping arcs between curves 0.68. The reviewer's concern is valid for missing or extra curves. Because the reward averages over curves, a single dropped, duplicated or spurious curve in a 5–8-curve scene still scores 0.80–0.83, and sets that preserve the correct containment tree are separated from sets that change it with AUROC of only 0.73. Three factors limit the practical impact:

  • The model gets the curve count right in 91–93% of held-out CurveBench scenes.
  • The answer reward has twice the weight of the geometric reward (λans=1.0λ_{ans} = 1.0 vs. λF=0.5λ_F = 0.5), so count errors that change the answer are still penalized.
  • A strict set reward, which gives 0 when the curve count is wrong and otherwise takes the worst-matched curve, raises the AUROC from 0.73 to 0.995 and is a direct fix.

Correlation with final-answer correctness. Across held-out scenes, rFr_F computed on the model's own trace predicts answer correctness:

ScenesAUROC (SFT / RL)Accuracy at rFr_F ≥ 0.8 / rFr_F < 0.2
Circuits (open, crossings)0.95 / 0.9799% / ≤ 10%
TRACER-Bench LT (open, many crossings)0.99 / 1.00100% / ≤ 8%
CurveBench (closed, multi-curve)0.85 / 0.8495–96% / 0%

AUROC stays at or above 0.88 on the open tasks across every stratum the reviewer names: number of crossings (0 to ≥ 4), clearance to the nearest curve, and number of curves. On CurveBench it ranges from 0.76 to 1.00 across strata, and its lower value has a specific cause: every wrong answer with rFr_F ≥ 0.8 had topologically correct traces. The error lies in reading the answer from the traces, not in the traces; rFr_F predicts whether the containment tree implied by the traces is correct with AUROC 1.000.

Correlation with human-assessed correctness. We manually examined 50 Tunnel Vision and TRACER-Bench traces, sampled across the range of rFr_F, without access to rFr_F, the model’s answer or the ground truth. A trace was judged correct only if it stays on the queried curve to its true end; the labels were then verified against the generator’s ground-truth path. rFr_F predicts the human judgement with AUROC 0.996 on Tunnel Vision and 0.979 on TRACER-Bench. Every trace with rFr_F ≥ 0.8 was judged correct, and none with rFr_F < 0.2.


We thank the reviewer again. We believe the controlled comparison, the empirical support for the theory and the reward analysis address all three concerns, and we would be glad to answer any further questions.

Dear Reviewer T4cQ,

Thank you again for your review. As today is the last day of the discussion period, we wanted to follow up on our response. If anything remains unclear, we are happy to clarify before the deadline. If our responses have addressed your concerns, we would appreciate it if you would consider raising your score.

Best regards,

Authors

I am raising my rating from 2 to 3. The matched answer-then-trace control addresses my main concern about attribution. With the same trace labels and output length, generating the trace before the answer improves SFT accuracy by 19.4 points on Tunnel Vision and 27.4 points on CurveBench. That is convincing evidence that conditioning on the trace helps on these tasks before the geometric reward is introduced. I view Theorem 12 as a conditional explanation for the studied connectivity setting. The reward analysis also addresses my question by quantifying where the current reward can give high scores to errors, especially near-parallel branches and missing or extra closed curves. These are limitations to document alongside the stronger SFT result.

Reviewer_Qn3aOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Chain-of-Coordinates (CoC): a VLM writes a sequence of image coordinates as text to trace a curve, either as the final output (drawing, handwriting) or as an intermediate step before answering connectivity and containment questions. The theory shows that a constructed transformer can trace simple paths and learn the rule from short examples, bounds the Fréchet error of the trace, and proves that tracing solves an NC¹-complete image connectivity family that direct answering cannot (unless TC⁰ = NC¹). Empirically, a separate Qwen3.5-2B is trained per task with SFT and then GRPO with a Fréchet-based reward. The paper reports that it beats direct-answer SFT on the four reasoning tasks and matches frontier models such as GPT-5.6 Sol.

Critical strengths

1.The method offers a simple, unified representation for several tasks. It handles drawing, writing, and connectivity questions through the same output format, without architectural changes, external tools, or image generation. The Fréchet-based reward is a natural fit for this representation, as it compares trajectories without requiring point-by-point correspondence.

2.Tracing helps most when color cues are unavailable. CoC SFT outperforms direct-answer SFT by 23/125 items on Tunnel Vision and 13/45 on CurveBench. On Tunnel Vision, the gap narrows from 18.4 to 2.4 percentage points when wire colors reveal the connections, suggesting that tracing is particularly useful when models need to follow the paths.

Critical weaknesses

1.The experiments do not clearly isolate the contribution of CoC. Results use a single seed and small test sets (roughly 19–125 items), with RL adding only 1–4 correct answers per task, leaving the reliability of these gains unclear. Direct-answer SFT + GRPO, GRPO without the Fréchet reward, and answer-then-trace SFT would help separate the effects of RL, reward design, and tracing before answering. The frontier comparison also needs qualification: the 2B models receive task-specific training, while Table 1 reports direct-answer frontier baselines, despite GPT-5.6 Sol improving from 84.4% to 95.6% on CurveBench with CoC prompting.

2.The connection between the theory and the empirical results remains unclear. Theorems 2–3 assume binary images, adjacent-pixel moves, and no crossings; the paper does not establish how their conclusions extend to the evaluated tasks. Theorem 12 builds on the S₅ separation of Li et al. (2024), supporting the value of intermediate computation but not establishing a particular advantage for coordinate traces. An empirical test of the theory’s implications would make its contribution more convincing.

Questions for authors
  1. What are the exact test-set sizes for each task? Could you report confidence intervals and results across training seeds?
  2. What value of τ is used, and what are the RL training-set size and number of optimization steps?
  3. Could you compare against direct-answer SFT + GRPO and CoC SFT + GRPO without the Fréchet reward to isolate the effects of RL and reward design?
  4. Does CoC still outperform answer-then-trace SFT when both use the same trace supervision?
  5. How often does the trace endpoint agree with the final answer, and does this consistency predict answer correctness?
3 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank the reviewer for the detailed and constructive review, and for recognizing that CoC offers a simple, unified representation for drawing, writing and connectivity reasoning, with the Fréchet reward as a natural fit. We address each point below.


Q1 & W1. Test-set sizes, confidence intervals and seeds

Test-set sizes are Quick, Draw! 50, Omniglot 50, BabyVision (Connect the Lines) 19, TRACER-Bench LT 50, Tunnel Vision (Circuit Connections) 125 and CurveBench 45. The reasoning tasks use their official test sets, and all evaluations use one greedy generation per example. We report 95% Wilson intervals over test items, paired bootstrap intervals for the differences and exact McNemar tests, and we retrained both models with 3 independent seeds:

Task (n)Direct-answer SFTCoC SFTDifferenceMcNemar p3 training seeds: direct vs. CoC
Tunnel Vision (125)75.2 [67.0, 81.9]93.6 [87.9, 96.7]+18.4 [+10.4, +27.2]6.6 × 10⁻⁵74.9 ± 2.8 vs. 93.3 ± 1.2
CurveBench (45)46.7 [32.9, 60.9]75.6 [61.3, 85.8]+28.9 [+13.3, +44.4]0.00245.9 ± 3.4 vs. 77.0 ± 2.6
TRACER-Bench (50)84.0 [71.5, 91.7]92.0 [81.2, 96.8]+8.0 [−4.0, +20.0]0.3482.7 ± 4.2 vs. 91.3 ± 1.2
BabyVision (19)0.0 [0.0, 16.8]5.3 [0.9, 24.6]+5.3 [0.0, +15.8]1.00.0 ± 0.0 vs. 7.0 ± 3.0

On Tunnel Vision and CurveBench, the gain is large, significant and stable: every CoC seed outperforms every direct-answer seed. On TRACER-Bench, CoC is ahead in every seed, but the gain is smaller and not significant for a single run, consistent with our color-shortcut analysis (§5.3). BabyVision's official test set has only 19 questions, so a single question changes accuracy by 5.3 pt. We agree this comparison is within noise and draw no conclusion from it on its own.

RL gains. We agree with the reviewer. RL adds 1–4 correct answers per reasoning task (+1 on Tunnel Vision, TRACER-Bench and BabyVision; +4 on CurveBench) and improves trace fidelity on both drawing tasks (Fréchet distance: Quick, Draw! 0.266 → 0.239; Omniglot 0.135 → 0.121). The improvement is consistent in direction across all six tasks, but no individual difference is significant at these test-set sizes (p ≥ 0.125). The main evidence for CoC is therefore the SFT-stage comparison above, which is significant and stable across seeds; RL acts as a consistent refinement on top of it.


Q2. RL details

  • τ = 100, with Fréchet distance computed on the 0–1000 coordinate grid: the reward falls to e⁻¹ when the trace deviates by 10% of the image side.
  • RL training set: 3,000 examples per task.
  • Optimization: 120–300 optimizer steps depending on the task, using GRPO with G = 8 rollouts per prompt at temperature 1.0. Training uses a learning rate of 5 × 10⁻⁶, a per-device batch size of 4 with 4 gradient-accumulation steps, and a 3,072-token completion budget (Appendix E).

Q3 & W1. Effects of RL and reward design

We ran both suggested controls, with all other settings unchanged: direct-answer SFT + GRPO (answer and format rewards only), and CoC SFT + GRPO without the Fréchet reward (λ_F = 0).

MethodTunnel VisionCurveBenchTRACER-Bench
Direct-answer SFT75.246.784.0
+ GRPO (answer and format rewards)76.051.186.0
CoC SFT93.675.692.0
+ GRPO without Fréchet reward (λ_F = 0)93.677.892.0
+ GRPO with Fréchet reward (full)94.484.494.0

Effect of RL. RL gives direct answering only small gains, and direct-answer GRPO remains far below CoC SFT without any RL (−17.6 pt on Tunnel Vision, −24.5 pt on CurveBench). RL therefore does not explain CoC’s advantage, which comes from the representation itself.

Effect of the Fréchet reward. Without the Fréchet reward, CoC RL stays close to SFT (+0, +1 and +0 correct answers), whereas with it RL adds +1, +4 and +1. The geometric reward therefore provides a richer training signal than the binary answer reward alone: within a group of rollouts that all reach the same answer, it still distinguishes better traces from worse ones, and this is where RL’s refinement comes from. As with the overall RL gains, the difference is directional at these test-set sizes.

AuthorsOpenReview ↗

Author Response (Part 2/2)

Q4. CoC vs. answer-then-trace with the same supervision

We trained an answer-then-trace model with exactly the same trace annotations and output length as CoC. It generates the answer before the trace, so the answer cannot depend on the trace. All models use the same images, base model and training budget (3 seeds):

MethodTrace labelsTrace before answerTunnel VisionCurveBenchTRACER-Bench
Direct-answer SFT✗✗74.9 ± 2.845.9 ± 3.482.7 ± 4.2
Answer-then-trace SFT✓✗73.9 ± 1.749.6 ± 5.185.3 ± 4.2
CoC SFT✓✓93.3 ± 1.277.0 ± 2.691.3 ± 1.2

Answer-then-trace performs like direct answering, within 1–4 pt on every task (p ≥ 0.36, Welch's t-test over seeds), and far below CoC on Tunnel Vision (−19.4 pt, p = 1 × 10⁻⁴) and CurveBench (−27.4 pt, p = 0.004). The trace annotations and extra tokens bring almost no benefit on their own; the gain comes from conditioning the answer on the trace.


Q5. Agreement between trace and answer

We compare each model’s written answer with the answer implied by its own trace on the Tunnel Vision task.

ModelAnswer agrees with traceAccuracy implied by traceAnswer accuracy
CoC SFT99.1%93.0%93.0%
CoC SFT + RL99.1%94.8%94.8%

The answer almost always follows the trace (114/115 for both models), so accuracy is determined by tracing: errors come from tracing mistakes, not from misreading a correct trace. Agreement is likewise 99.1% on the Unique-Colors variant.


W2. Connection between theory and experiments

Theorems 2–3. These establish sufficient conditions in the simplest setting, and the paper extends them toward the evaluated tasks:

  • Crossings. Proposition 5 covers crossings and junctions that can be resolved from local context.
  • Pixel values. Proposition 5 allows any finite pixel alphabet, including 8-bit RGB.
  • Coordinates. Proposition 7 and Corollary 10 relate adjacent-pixel paths to the sampled, rounded coordinates used in training.

Theorem 12. We agree that the separation builds on the S₅ construction (Barrington, 1989; Li et al., 2024) and shows the value of intermediate computation in general, and we do not claim coordinates are the only representation that achieves it. Our contribution is to show that coordinate traces suffice, with AC⁰ computation per token.

Empirical tests of the theory's implications. Theorem 12 makes three predictions that can be tested on trained VLMs:

  1. The advantage comes from intermediate computation before the answer. In Theorem 12, CoC succeeds because the answer is computed from the coordinates generated before it. If this is the mechanism, generating the same trace after the answer should not help. This is what we observe: answer-then-trace, trained with the same supervision, performs like direct answering, far below CoC (Q4). Conversely, the CoC model's answer follows its own trace endpoint in 99.1% of cases (Q5).
  2. The advantage appears when connectivity requires sequential tracing. The separation holds because the curve must be followed step by step. When the connection can be read off directly, there is nothing to trace and the advantage should vanish. With identical supervision, CoC's gain is +18.4+18.4 pt on Single-Color Tunnel Vision (p=6.6×10−5p = 6.6 \times 10^{-5}) but only +2.4+2.4 pt on Unique Colors, where wire colors reveal the connections (p=0.45p = 0.45).
  3. The advantage costs tokens that grow with path length. Theorem 12(ii) uses O(Nimglog⁡Nimg)O(N_{\mathrm{img}} \log N_{\mathrm{img}}) tokens. Measured cost likewise grows with path length, from 74 tokens per question on Tunnel Vision to about 1,270 on BabyVision's long, dense paths.

We thank the reviewer again. The suggested experiments made our claims more precise and better supported, and we would be glad to answer any further questions.

Thank you for the detailed response. I’ll keep my positive score.

Reviewer_WC7FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Chain-of-Coordinates (CoC), a textual representation in which VLMs autoregressively generate ordered coordinate sequences to trace curves, requiring no architectural changes, tools, or intermediate image generation. The authors provide a theoretical pipeline covering representability, learnability with length generalization, Fréchet-distance error bounds, and a conditional TC⁰/NC¹ separation showing CoC's computational advantage over direct answering. A two-stage SFT+GRPO framework is proposed, with a continuous-Fréchet geometric reward that tolerates different samplings and curve parameterizations. Across six tracing/connectivity/containment tasks, CoC training outperforms direct-answer SFT, and a 2B model with RL matches several frontier models. Additional analyses clarify when CoC helps and when prompting alone suffices.

Critical strengths
  1. CoC unifies visual intermediate states as plain text coordinates across both output and reasoning tasks; the four-layer theory (representability, learnability, error bounds, computational separation) is rare and rigorous for this line of work.
  2. The continuous Fréchet reward handles sampling invariance, closed-curve start/direction ambiguity, and multi-curve Hungarian matching, directly addressing the "one curve, many valid traces" problem that SFT cannot.
  3. Insightful "when it helps" analysis. The single-color vs. unique-color experiment and the linking of prompting gains to underlying tracing ability are valuable findings.
Critical weaknesses
  1. CoC applies only to tasks reducible to explicit curve tracing on stylized synthetic images (line following, wiring, nested curves); its applicability to mainstream visual reasoning—charts, documents, GUIs, natural scenes—is neither demonstrated nor discussed concretely. The contribution is a specialized technique for one task family rather than the claimed general paradigm for "connecting visual perception and reasoning."
  2. Key comparisons (e.g., 0.053 vs 0.000 on BabyVision) rest on tiny test sets (~50 examples for some tasks) with no error bars or significance tests; several conclusions may be within noise.
  3. The theory relies on idealized assumptions (binary images, exact arithmetic, unlimited context), and the token/latency cost of long coordinate sequences—a central price term in Theorem 12—is not quantified empirically.
Questions for authors

See the weaknesses.

5 comments
AuthorsOpenReview ↗

Author Response (Part 1/2)

We thank the reviewer for the careful and constructive review, and for recognizing the theory, the Fréchet reward and the "when it helps" analysis. We address each weakness below.


W1. Scope

We agree that CoC is not intended as a universal solution to all visual reasoning tasks. Our main contribution is to establish explicit coordinate generation as a general computational mechanism for visual reasoning over geometric and topological structure, supported by both theoretical analysis and empirical evidence.

Within this scope, the evaluation covers two distinct settings rather than a single tracing task:

  • CoC as output, where the trace is the answer: drawing (Quick, Draw!) and handwriting (Omniglot).
  • CoC as reasoning, where the trace supports a separate answer: endpoint matching (BabyVision, TRACER-Bench), circuit connectivity (Tunnel Vision) and hierarchical containment (CurveBench).

The evaluation is also not limited to stylized synthetic data. The drawing and writing datasets use real human pen strokes. The reasoning models are evaluated on official benchmark test sets (§5.1). This type of task remains a weakness of frontier VLMs: on Tunnel Vision, the best frontier model reaches 55.2%, while our 2B CoC model reaches 93.3 ± 1.2% over 3 training seeds.

Applications. Many mainstream visual reasoning tasks require keeping track of spatial state along a structure: where a line goes, what an arrow connects, or which region a boundary encloses. In a flowchart, finding the step after a given box requires following its outgoing arrow. On a map or in a photo of tangled cables, deciding what connects to what is endpoint matching, as in TRACER-Bench and Tunnel Vision. In GUIs, many actions are trajectories rather than single clicks: drawing or signing on a canvas, lasso selection around an object, dragging a slider or handle along a track, and drawing a route on a map. For these, the coordinate trace is itself the action. Tasks that depend mainly on semantics or text, such as document QA, are outside CoC’s scope; there CoC would complement rather than replace existing methods.

CoC as a computer-use action. Drawing in computer use requires a continuous trajectory rather than a single click. CoC is a natural fit for computer-use interfaces because both are polylines. A CoC is, by construction, a sequence of points joined by straight segments (§3). Computer-use drag actions work the same way: a drag is specified as an ordered list of points, and the cursor moves in straight segments between them; these interfaces have no curve primitive. OpenAI's computer tool, for example, expresses a drag as an ordered path of coordinates. A CoC therefore maps one-to-one onto a drag action, with no curve fitting or resampling. Curve representations such as splines or pixel masks would first have to be converted into such a path.

To test this, we embedded the Quick, Draw! test set (n = 50, held-out categories) in a canvas-drawing task. The model sees a screenshot of a paint application showing the reference sketch and is asked to reproduce it on a blank canvas.

Starting from our CoC model, we fine-tuned it with the same recipe as §5.2 to output its CoC for each stroke, followed by the matching drag action:

<CoC>(x1,y1),(x2,y2),…,(xk,yk)</CoC>
drag(path=[(x1,y1),(x2,y2),…,(xk,yk)])

The drag path is the CoC itself. Both are the same ordered sequence of points joined by straight segments, so the model does not need to learn a new representation for acting. It only learns to emit the trace it already produces in the action format. This is why adapting a CoC model to computer use takes only lightweight fine-tuning.

The drag is executed on the OS cursor in a sandboxed virtual desktop via xdotool (mouse down at the first point, move through each later point, mouse up at the last), following OpenAI’s reference integration. We read the drawn strokes back from the canvas and computed their Fréchet distance to the reference (normalized as in Table 1):

MethodOutputFréchet ↓
GPT-5.6drag(path=[…])0.284
Qwen3.5-2B + CoC SFT (ours)CoC + drag(path=[…])0.260

Our 2B model draws more accurately than GPT-5.6, which shows that the same representation applies directly to GUI drawing.

AuthorsOpenReview ↗

Author Response (Part 2/2)

W2. Error bars and significance tests

Confidence intervals, significance tests and seeds. We report 95% Wilson intervals over test items, paired bootstrap intervals for the differences, and exact McNemar tests on the same items. We also retrained both models with 3 independent seeds:

Task (n)Direct-answer SFTCoC SFTDifferenceMcNemar p3 training seeds: direct vs. CoC
Tunnel Vision (125)75.2 [67.0, 81.9]93.6 [87.9, 96.7]+18.4 [+10.4, +27.2]6.6 × 10⁻⁵74.9 ± 2.8 vs. 93.3 ± 1.2
CurveBench (45)46.7 [32.9, 60.9]75.6 [61.3, 85.8]+28.9 [+13.3, +44.4]0.00245.9 ± 3.4 vs. 77.0 ± 2.6
TRACER-Bench (50)84.0 [71.5, 91.7]92.0 [81.2, 96.8]+8.0 [−4.0, +20.0]0.3482.7 ± 4.2 vs. 91.3 ± 1.2
BabyVision (19)0.0 [0.0, 16.8]5.3 [0.9, 24.6]+5.3 [0.0, +15.8]1.00.0 ± 0.0 vs. 7.0 ± 3.0
  • Tunnel Vision and CurveBench: the gain is large, significant and stable. Every CoC seed outperforms every direct-answer seed.
  • TRACER-Bench: CoC is ahead in every seed (+8.7 pt on average). This matches our analysis in §5.3: when paths have distinct colors, connections can be read off without tracing. The Single-Color vs. Unique-Color comparison makes the same point (+18.4 pt, p = 6.6 × 10⁻⁵, vs. +2.4 pt, p = 0.45).
  • BabyVision: we agree that this comparison is noisy. The official Connect-the-Lines test set contains only 19 questions, which limits what this benchmark can show. Across 3 seeds, CoC solves 1–2 items and direct answering none: the direction is consistent but not significant. We therefore treat BabyVision as a hard case and not as evidence for CoC.

W3. Idealized assumptions and token/latency cost

Assumptions. We agree that the theory relies on idealized assumptions. To state them precisely:

  • Context. The theory does not require unlimited context, but it does require context that grows polynomially with image size (§4.1). Theorem 12(ii) uses O(Nimglog⁡Nimg)O(N_{img}\log N_{img}) tokens. We measured how much context CoC actually uses (below).
  • Arithmetic. Coordinates are integers on a finite 0–1000 grid, written exactly as tokens (Lemma 1), and rounding error is part of our bounds (Prop. 7, Cor. 10).
  • Binary images. Binary images are used for clarity of presentation. The analysis depends on pixel values only through fixed-size local patches, and Proposition 5 already covers any finite pixel alphabet, including 8-bit RGB. The results therefore extend to colored images whenever curve membership can be determined locally. Images that need global or semantic context to identify a curve are outside our scope.

Our experiments drop these assumptions (RGB inputs and a 4,096-token context), and the results follow the theory's prediction: CoC helps most when connectivity must be traced (Single-Color vs. Unique-Color, above).

Token and latency cost. Mean tokens and time per question:

TaskDirect answer: tokens / timeCoC: tokens / timeFrontier models: tokens, fewest … most
Tunnel Vision8 / 0.7 s74 / 3.8–4.3 s331 (GPT-5.6 Sol, direct) … 5,933 (Gemini 3.8 Flash, CoC prompting)
TRACER-Bench4 / 0.8 s100–114 / 4–5 s50 (GPT-5.6 Sol) … 616 (Gemini 3.8 Flash)
CurveBench28 / 3.5 s870–910 / 34–42 s201 (Claude Opus 5, direct) … 5,536 (Gemini 3.1 Pro, CoC prompting)
BabyVision21 / 2.6 s~1,270 / 48 s599 (Claude Opus 5, direct) … 14,483 (Gemini 3.8 Flash, CoC prompting; hit the token cap)
Quick, Draw!—~455 / ~29 s1,226 (GPT-5.6 Sol) … 7,771 (Gemini 3.1 Pro)
Omniglot—~188 / ~8 s906 (GPT-5.6 Sol) … 5,641 (Gemini 3.1 Pro)

Our times are measured with Hugging Face generate at batch size 1 on one H100 (about 20–30 tokens/s). This uses the unoptimized evaluation code; with a serving engine, time scales with token count. Frontier counts are completion tokens and include hidden reasoning.

  • Compared with direct answering, CoC uses 9–60× more tokens, and that is what buys its accuracy gains. On Tunnel Vision, 66 extra tokens raise accuracy by 18.4 pt; on CurveBench, about 860 extra tokens raise it by 31.1 pt. The cost grows with path length, as expected from the O(Nimglog⁡Nimg)O(N_{img}\log N_{img}) term in Theorem 12.

  • Compared with frontier models, CoC uses fewer tokens than any frontier model on Tunnel Vision, Quick, Draw! and Omniglot, and lies within the frontier range on the other tasks. On Tunnel Vision, frontier models spend 331–5,933 tokens per question and reach at most 55.2%, while CoC spends 74 and reaches 93.3%.


We thank the reviewer again. We believe the clarified scope, the computer-use experiment, the statistical analysis and the cost measurements address all three concerns, and we would be glad to answer any further questions.

Dear Reviewer WC7F,

Thank you again for your review. As today is the last day of the discussion period, we wanted to follow up on our response. If anything remains unclear, we are happy to clarify before the deadline. If our responses have addressed your concerns, we would appreciate it if you would consider raising your score.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.