- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Chain-of-Coordinates (CoC), a representation that enables vision-language models (VLMs) to express curves, paths, and intermediate spatial states through textual coordinate sequences. The paper provides theoretical analyses of representability, local-rule learnability, geometric approximation error, and the computational complexity of endpoint connectivity. It further introduces a two-stage SFT+GRPO training framework with a geometric reward based on continuous Fréchet distance. Experiments across six tasks involving drawing, writing, line connection, nonsemantic tracing, circuit connectivity, and curve containment demonstrate improvements in tracing quality and visual reasoning performance. However, these benefits depend on the availability of direct visual shortcuts and the model's underlying tracing capability.
Critical strengths
Geometrically meaningful intermediate representation. Coordinate sequences serve as both executable outputs and intermediate reasoning states. The continuous Fréchet reward accommodates different sampling densities, providing a more appropriate measure of trajectory similarity than strict point-wise correspondence.
Connection between theoretical analysis and empirical validation. The analyses of local-rule learnability, geometric error decomposition, and endpoint connectivity provide theoretical support beyond empirical intuition. Experiments across six tasks further demonstrate that CoC is particularly beneficial when direct visual shortcuts, such as distinctive colors or appearance cues, are unavailable.
Critical weaknesses
Insufficiently controlled attribution of performance gains. Compared with direct-answer SFT, CoC training benefits from additional trajectory annotations and longer output sequences, as well as an additional geometric reward during RL. The current comparisons do not adequately disentangle whether the improvements arise from the coordinate-based reasoning representation itself or from richer trajectory supervision and additional training signals.
Limited practical implications of the computational complexity analysis. Theorem 12 relies on an artificially constructed family of block-based curves, specific AC0/TC0/NC1 computational assumptions, and the conditional separation TC0 ≠ NC1. These theoretical results do not directly establish that CoC provides superior reasoning capabilities over direct answering in real-world VLMs.
Questions for authors
Reliability of the Fréchet-based geometric reward. The reward allows arbitrary starting points and traversal directions for closed curves, as well as Hungarian matching for multiple curves. Could these invariances assign high rewards to geometrically similar but topologically incorrect trajectories, particularly in cases involving multiple curves, crossings, or incorrect curve correspondences? Please provide an analysis of the correlation between geometric rewards and final-answer correctness or human-assessed trajectory correctness, along with sensitivity analyses under these challenging conditions.
4 comments
Author Response (Part 2/2)
Q1. Reliability of the Fréchet-based geometric reward
We address each failure mode the reviewer raises below, then report the correlation analyses requested. In short, the reward reliably penalizes wrong turns and wrong curves; the reviewer's concern holds for two specific cases, which we quantify.
1. Start-point and direction invariance. This cannot admit a topologically incorrect trace. The invariance applies only to closed curves, where every start point and direction traces exactly the same set. Open reasoning traces (Tunnel Vision, TRACER-Bench) are scored in a fixed direction.
2. Crossings. Wrong turns are heavily penalized. Continuous Fréchet distance is the maximum separation along the best traversal, not an average, so a trace that takes the wrong branch is penalized by the full distance between the branches. Measured on open traces: correct traces with small vertex noise (σ ≤ 20 grid units) keep a median of 0.68–0.92, while a branch switch at a crossing drops it to 0.01–0.03. Correct and incorrect traces are separated with AUROC 0.992 on Tunnel Vision and 1.000 on TRACER-Bench.
Exception: near-parallel curves (as seen in Tunnel Vision). When a neighbouring wire runs 25–50 grid units away, 35% of jumps onto it keep ≥ 0.5.
3. Incorrect curve correspondence. For queried curves this cannot occur. In Tunnel Vision and TRACER-Bench, each queried trace is compared with its own reference curve, with no Hungarian matching. Tracing the wrong curve is therefore penalized: a jump to the nearest neighbouring curve drops to 0.01–0.15.
4. Multiple curves with Hungarian matching (CurveBench). Errors within the curves are penalized: merging two curves gives 0.61, and swapping arcs between curves 0.68. The reviewer's concern is valid for missing or extra curves. Because the reward averages over curves, a single dropped, duplicated or spurious curve in a 5–8-curve scene still scores 0.80–0.83, and sets that preserve the correct containment tree are separated from sets that change it with AUROC of only 0.73. Three factors limit the practical impact:
- The model gets the curve count right in 91–93% of held-out CurveBench scenes.
- The answer reward has twice the weight of the geometric reward ( vs. ), so count errors that change the answer are still penalized.
- A strict set reward, which gives 0 when the curve count is wrong and otherwise takes the worst-matched curve, raises the AUROC from 0.73 to 0.995 and is a direct fix.
Correlation with final-answer correctness. Across held-out scenes, computed on the model's own trace predicts answer correctness:
| Scenes | AUROC (SFT / RL) | Accuracy at ≥ 0.8 / < 0.2 |
|---|---|---|
| Circuits (open, crossings) | 0.95 / 0.97 | 99% / ≤ 10% |
| TRACER-Bench LT (open, many crossings) | 0.99 / 1.00 | 100% / ≤ 8% |
| CurveBench (closed, multi-curve) | 0.85 / 0.84 | 95–96% / 0% |
AUROC stays at or above 0.88 on the open tasks across every stratum the reviewer names: number of crossings (0 to ≥ 4), clearance to the nearest curve, and number of curves. On CurveBench it ranges from 0.76 to 1.00 across strata, and its lower value has a specific cause: every wrong answer with ≥ 0.8 had topologically correct traces. The error lies in reading the answer from the traces, not in the traces; predicts whether the containment tree implied by the traces is correct with AUROC 1.000.
Correlation with human-assessed correctness. We manually examined 50 Tunnel Vision and TRACER-Bench traces, sampled across the range of , without access to , the model’s answer or the ground truth. A trace was judged correct only if it stays on the queried curve to its true end; the labels were then verified against the generator’s ground-truth path. predicts the human judgement with AUROC 0.996 on Tunnel Vision and 0.979 on TRACER-Bench. Every trace with ≥ 0.8 was judged correct, and none with < 0.2.
We thank the reviewer again. We believe the controlled comparison, the empirical support for the theory and the reward analysis address all three concerns, and we would be glad to answer any further questions.
Dear Reviewer T4cQ,
Thank you again for your review. As today is the last day of the discussion period, we wanted to follow up on our response. If anything remains unclear, we are happy to clarify before the deadline. If our responses have addressed your concerns, we would appreciate it if you would consider raising your score.
Best regards,
Authors
I am raising my rating from 2 to 3. The matched answer-then-trace control addresses my main concern about attribution. With the same trace labels and output length, generating the trace before the answer improves SFT accuracy by 19.4 points on Tunnel Vision and 27.4 points on CurveBench. That is convincing evidence that conditioning on the trace helps on these tasks before the geometric reward is introduced. I view Theorem 12 as a conditional explanation for the studied connectivity setting. The reward analysis also addresses my question by quantifying where the current reward can give high scores to errors, especially near-parallel branches and missing or extra closed curves. These are limitations to document alongside the stronger SFT result.
Author Response (Part 1/2)
We thank the reviewer for the thoughtful review, and for recognizing CoC's role as executable output and reasoning state, and the connection between our theory and experiments. We address each point below.
W1. Attributing the performance gains
We agree that the representation should be separated from the extra supervision and training signal. The reviewer names three possible sources of the gain: trajectory annotations, longer outputs, and the geometric reward.
Trajectory annotations and output length. We trained an answer-then-trace model with exactly the same trace annotations and output length as CoC. It generates the answer before the trace, so the answer cannot depend on the trace. All models use the same images, base model and training budget (mean ± std over 3 training seeds):
Answer-then-trace is trained on the matching trace annotations and compute, and produces outputs of the same length as CoC, so any benefit of the extra supervision, including its use as an auxiliary task that improves the learned features, is available to both models. The only difference is the order: because generation is autoregressive, the answer cannot be computed from a trace produced after it. Comparing the two therefore separates the effect of the extra supervision from the effect of reasoning through the trace. Answer-then-trace performs close to direct answering on every task (p≥0.36, Welch's t-test over seeds) and far below CoC on Tunnel Vision (−19.4 pt, p=1×10−4) and CurveBench (−27.4 pt, p=0.004). With matched data and compute, the trace annotations and extra tokens therefore bring little on their own; the gain comes from the additional representation power that CoC provides by computing the answer through the trace, which is empirical evidence for the practical relevance of Theorem 12.
Geometric reward. All results in the table are measured after SFT, before any RL. The improvement over direct answering therefore comes from the representation itself, not from the additional reward. RL with the geometric reward then refines the model: it improves trace fidelity (Fréchet distance on Quick, Draw!: 0.266 → 0.240) and adds further gains on Tunnel Vision, TRACER-Bench and BabyVision.
Supporting evidence. Two existing results lead to the same conclusion. First, on Tunnel Vision Circuit Connections the trace supervision is identical in the Single-Color and Unique-Color settings, yet CoC’s advantage drops from +18.4 pt (p = 6.6 × 10⁻⁵) to +2.4 pt (p = 0.45) once colors reveal the connections. The gain follows whether tracing is needed, not how much supervision is given; this also explains the smaller gap on TRACER-Bench, where paths often have distinct colors. Second, CoC helps even with no trace supervision at all: CoC prompting alone improves frontier models (e.g. GPT-5.6 Sol on CurveBench, 80.0 → 95.6; Table 2).
W2. Practical implications of Theorem 12
We agree that Theorem 12 is a conditional result on a constructed family. Its purpose is to explain the mechanism behind CoC's advantage, not to guarantee performance. Direct answering has to resolve global connectivity within a fixed number of parallel steps, whereas CoC carries spatial state forward one segment at a time. The separation builds on the S₅ construction of Barrington (1989) and Li et al. (2024). Our contribution is to show that coordinate traces suffice for this separation with AC⁰ computation per token.
The theory makes three predictions that our results on trained VLMs support:
We therefore present the theory as an explanation of why and when CoC helps, consistent with these results, rather than as a guarantee for every real-world setting.