acceptodds
Under review as a conference paper at ICLR 2027

Visual Chain of Thought as a Resource-Bounded Machine

Abstract

Multimodal models now produce intermediate images and read them back through a connector. Text chain of thought theory counts only tokens, so the visual loop therefore lacks a formal account. In this paper, we model visual chain of thought as a resource bounded machine with two operators. A paint writes a pixel grid through a constant depth local decoder, and a read returns a bounded number of tokens through a connector encoder. Every bound is governed by three quantities, the pixels written at each step, the read bandwidth of the connector, and the state entropy the next step must observe. Storing one incompressible working image in text requires a quadratic number of tokens, while a single paint produces the same image in one step. Because a local decoder of constant radius implements one cellular automaton update, source to target connectivity on an open grid admits a linear number of paints, whereas a packed parent pointer transcript remains faster than linear in the diameter. When the read bandwidth falls below the state entropy, native regeneration cannot reconstruct the tape, and answers on the language side no longer depend on the image, so the intermediate image is decorative. The same bound determines what to draw, and the working image is a sufficient statistic of the state at low entropy. Frozen open weight vision language models reproduce this separation on low entropy queries. Accuracy stays at the text baseline until the connector retains enough tokens, and a corrupted render or token stream changes the answer. Larger models cross the same threshold at a smaller token budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual chain of thought, Circuit complexity, Multimodal reasoning, Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2f7dOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a resource-bounded model of visual chain of thought with a local parallel Paint operator, a bandwidth-limited Read operator, and a flag indicating whether pixels persist between rounds. It studies image materialization costs, cellular-automaton computation, state reconstruction through a connector, and the dependence of language-side answers on visual states. The proposed framework is evaluated using scripted cellular-automaton/BFS constructions, reconstruction experiments, and token-retention and intervention studies with frozen Qwen3-VL models.

Critical strengths
  1. The framework identifies useful architectural distinctions. Separating write arity, read bandwidth, and persistent pixel storage provides a useful way to reason about visual working memory. In particular, the distinction between the information needed to update a canvas and the information needed to report a final answer is relevant to visual reasoning systems.

  2. The experiments include informative interventions. Table 13 studies both pixel corruption and token replacement at fixed pixels. These interventions provide stronger evidence about whether the model uses the visual-token path than accuracy or textual references to the image alone.

Critical weaknesses
  1. The principal computational lower bound does not follow from the stated restrictions. Theorem 4 counts the nodes in an unrolled cellular-automaton computation and concludes that every text machine naming at most BB cell-layer pairs per step requires Ω(t3/B)\Omega(t^3/B) steps. However, restricting the output names does not require an algorithm to materialize that particular computation graph.

    Consider the radius-one binary rule that replaces each cell by the AND of its 3×33\times3 neighborhood. This rule is nonlinear over GF(2)\mathrm{GF}(2), and the queried output is sensitive to every initial cell in its light cone. Nevertheless, after tt updates, the output is simply

    xt(q)=⋀∥u−q∥∞≤tx0(u).x_t(q)=\bigwedge_{\|u-q\|_\infty\le t}x_0(u).

    A constant-depth threshold computation can evaluate this conjunction directly; Appendix C.1 explicitly permits conjunction over a marked input subset. The result can be emitted with one token naming the queried cell-layer pair. Thus, sensitivity and nonlinearity do not establish the claimed cubic lower bound.

    The copy-north example in Lemma F.1 and Table 5 is also problematic: a queried output depends on one initial cell, rather than all nodes in the counted square cone. A lower bound for a prescribed materializing simulation would be substantially narrower than the current theorem.

  2. The information-theoretic arguments do not establish the central residual and decoration claims. Theorem 8 assumes that an open-grid BFS state has entropy Θ(m2)\Theta(m^2). On an obstacle-free grid, however, the visited/frontier configuration at round jj is determined by the source coordinates and jj, through Manhattan distance. At a fixed round, varying only the source produces at most m2m^2 configurations, so

    H(Sj)≤log⁡2(m2)=2log⁡2m.H(S_j)\le\log_2(m^2)=2\log_2m.

    Conditional on the source and round, its entropy is zero. This invalidates the entropy premise used for the paired separation.

    Theorem 7 also moves from an observational information bound to an interventional conclusion without sufficient assumptions. Let Txt=R=Y\mathrm{Txt}=R=Y observationally, and let the answer policy be A:=RA:=R. Then

    H(A∣Txt)=I(A;R∣Txt)=0,H(A\mid\mathrm{Txt})=I(A;R\mid\mathrm{Txt})=0,

    yet replacing RR by 1−Y1-Y flips every answer. Observational determinacy does not establish that the policy ignores the image.

    Finally, the Shannon-entropy assumption in Theorem 6 is insufficient for its universal 1/31/3 reconstruction-error guarantee. A distribution with probability 0.990.99 on one state and probability 0.010.01 spread uniformly over 2M2^M other states has entropy

    H(S)=h2(0.01)+0.01M,H(S)=h_2(0.01)+0.01M,

    where h2h_2 is binary entropy. Its entropy can be arbitrarily large, while a constant guess has error only 0.010.01. These issues affect the main theoretical conclusions rather than merely their presentation.

Questions for authors

The following clarification and correction requests could materially affect the assessment.

  1. Which explicit restriction on TextCoTB\mathrm{TextCoT}_B excludes the direct AND computation described above? If the intended class must materialize a prescribed computation graph, can the theorem be restated for that class? How should Lemma F.1 and Table 5 be interpreted for copy north?

  2. What probability distribution and conditioning give the open-grid BFS state in Theorem 8 entropy Θ(m2)\Theta(m^2)? How does the argument account for the source coordinates, round counter, and input tape already available to the machine?

  3. What additional distributional assumptions are required for Theorem 6, and what structural causal assumptions justify Theorem 7’s intervention claim? Please distinguish whole-state reconstruction, final-answer prediction, and interventional dependence on the image.

  4. The frozen protocol specifies a subset of Nv=196N_v=196 visual tokens, but Tables 12 and 13 report k=256k=256. What does kk represent in those rows, and how are budgets above NvN_v implemented? Which measurements directly compare residual and nonresidual iterative generation on the same task?

Reviewer_SbdzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper models visual chain of thought as a resource bounded machine. It uses two operators named paint and read. Paint writes a pixel grid through a local decoder. Read returns a bounded number of tokens through a connector encoder. The paper provides theoretical bounds on simulation cost, write arity, read bandwidth, and state entropy. It shows that visual chain of thought can reduce the number of steps through spatial parallelism. It also studies when intermediate images become decorative due to limited read bandwidth. In the experiments, the paper uses synthetic machines and frozen Qwen3-VL models. The experimental results show resource savings on grid tasks and bandwidth thresholds on simple visual queries.

Critical strengths
  1. The paper introduces a visual working tape with Paint, Read, and a residual flag, and it provides a theoretical framework to study the resource costs of visual reasoning.

  2. Some experiments are interesting. The paper studies when intermediate images actually affect the answer. Also the corrupted render and token intervention experiments provide interesting evidence about whether models use visual information.

Critical weaknesses
  1. There is one mathematical error. In page 26, the formula for the light cone count seems incorrect. For example, when t=1, the left side equals 10, while the right side equals 20. it raises concerns about the correctness of the theoretical analysis for me. Hope the authors can check and correct it.

  2. I'm confused about some therems. For example, in theorem 4, the paper assumes that all intermediate nodes must be computed or stored, but do not explain why this is necessary. Also in theorem 8, the paper claims that the visited map contains a large amount of information. However, BFS is deterministic, so the visited map can be determined from the input grid, source, and current step. It is therefore unclear why the model must recover the entire map from visual tokens. I think BFS is not a good example to justify the theorem 8. Could the authors give more discuss about it?

  3. In the paper, most experiments use synthetic machines with local rules. The frozen Qwen3-VL experiments mainly test simple colour queries. Connectivity stays near the text baseline at every budget. Therefore, the experiments do not clearly demonstrate that the multimodal models can benefit from the proposed visual working tape.

Questions for authors
  1. In theorem 8, why does the visited map on an obstacle free grid have a big entropy? Because given the input grid, source, and iteration step, the visited map is deterministic. Could the authors clarify the entropy assumption and its effect on the residual separation?

  2. In theorem 4, why must every light cone node be explicitly computed or stored by the text machine?

Reviewer_RdJUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The submission explores the effect of a visual work area on transformer expressivity, particularly with regards to computing source-target reachability. The main construction is a process for moving information back and forth between the text tape and visual pad, and then specification of the local operations allowed on the canvas. Information is tracked for two variants -- one where the visual pad is persistent and one where it must be drawn anew during every paint operation. Given certain restrictions on the setup, the authors chart out regimes of information processing such as where the visual pad plays no role vs boosting performance relative to text-based serial solutions of the reachability task.

Critical strengths

It's an interesting idea to account for the effect of a visual workspace on transformer expressivity, itself an active area of research of late. The particular construction is reasonable and seems like a step in the right direction.

Critical weaknesses

I found the presentation particularly laborious to parse, with movement from generality to specificity and then to ambiguity. Examples: Definitions 1-3 set up the visual working tape in full generality and then immediately after "A frozen vision transformer, probed linearly on a visited map, never reaches Hamming distance 0.05..." (L197), with no prior mention of the probing or even that the generality switch to the later reachability instantiation. The paragraph starting around L87 similarly migrates from general statements about expressivity to reachability-specific statements.

Presentation aside, ambiguity is a central weakness of the paper that stands in the way of any downstream claims. What exactly are the requirements for A1 -- that the encoder does not run the task? The qwen experiments have scant detail even when one searches the appendix -- how is the Paint operation implemented? What was the input to the linear probes?

Finally, the result around establishing when the visual workspace is "decoration" seems dubious. The first claim around when the text tape already contains all info about the answer (L98) is problematic -- subsequent processing can still depend on the content of the sketchpad. The later claim around Thm 7 jumps from incomplete information to zero effect, more explicitly in Appx G.

Questions for authors

What motivates the simple masking encoder, and how do the expressivity results depend on learning the bit-restricted communication channel back to the text tape? Can the authors provide much more specificity about the conditions in A1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.