- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether visual latent reasoning states contain useful derivations and influence answer prediction. It introduces CALR, which combines information-balanced optical compression, a frozen question-blind reader, and derivation supervision from a frozen OCR decoder. A parallel producer is subsequently trained to generate four dependent latent blocks. An optional internal reader supports unified inference. Experiments cover five mathematical reasoning benchmarks and several model families. The main configuration improves GSM8K accuracy from RoT's 37.8% to 63.8% with 32 latent states. Latent replacement, decoding, and intermediate-state substitution provide evidence about the resulting representations.
Critical strengths
-
The method addresses a concrete failure mode with a meaningful training change. Question-blind answer supervision removes the direct question-to-answer shortcut from the external supervisory route. Derivation supervision gives that route structured content. Table 1 reports a substantial 26.0-point GSM8K gain over RoT at the same latent count. Table 2 shows improvements across all twelve reported model-benchmark combinations. Table 4 also supports retaining external supervision during autoregressive refinement.
-
The causal diagnostics are unusually helpful. Tables 11 and 15 separately test answer-side reliance and dependence between generated blocks. Table 17 is especially valuable. Donors share the original final answer but differ in an intermediate quantity. Feedback substitution produces the corresponding counterfactual answer in 75% and 80% of the two sets. Reader-only substitution does so in only 0.8%. This provides meaningful evidence that intermediate information affects subsequent generation. It strengthens the contribution beyond simply decoding readable text from latents.
Critical weaknesses
-
The empirical tests do not fully establish the stated functional requirements. C1 in Section 2.1 concerns any change in the per-example answer distribution. Section 2.2 nevertheless declares that DLR-SFT fails C1 using aggregate accuracy, even though deletion changes accuracy from 53% to 51%. Aggregate accuracy cannot determine this distributional condition. C2 requires a valid complete derivation. Table 13 instead measures recovery of reference arithmetic steps, with a score of 0.74 for CALR. It does not report how often the full decoded derivation is valid or contains extra incorrect steps. Appendix E.5 acknowledges semantic and equation discrepancies in the transcriptions. Table 17 provides stronger causal evidence on selected cases, but does not resolve the full-dataset validity claim. The paper should distinguish these supported partial properties from satisfaction of C1 and C2 as formally defined.
-
The comparison is not sufficiently reconciled or controlled. Unified parallel CALR scores 52.4% in Tables 1 and 10, but the full parallel ablation scores 52.2% in Table 4. CALR's answer accuracy is 63.8% in Table 1 and 63.5% in Table 13. Appendix D.1 assigns these evaluations to the full GSM8K test set. Different seeds or checkpoints could explain these values, but that distinction is not documented. The meaning of the reported uncertainty and the number of runs are also unspecified. In addition, SIM-CoT is discussed but not evaluated. The paper needs a controlled comparison to isolate the value of the proposed anchoring design. Equal latent count alone does not match training resources or inference cost across the different generation procedures.
Questions for authors
-
Clarification: What operational threshold determines satisfaction of C1? Can you report paired prediction changes or distributional changes, rather than infer the formal condition from aggregate accuracy? For C2, what fraction of decoded trajectories are complete valid derivations, including checks for extra incorrect steps?
-
Clarification: How were the two 120-problem sets in Table 17 constructed? Were examples selected using native correctness or successful intervention outcomes? Please specify the eligible population, donor selection procedure, and how the substituted block was aligned with the targeted intermediate quantity.
-
Clarification: Please reconcile 52.4% versus 52.2% and 63.8% versus 63.5%. Which checkpoints, seeds, readout routes, and evaluation runs underlie these tables? What do the uncertainty values represent? The 300-example subset in Table 11 and the external-reader route in Table 14 already explain their different values; those are not the discrepancies at issue.
-
Evidence request: How does CALR compare with SIM-CoT under the same base model, training examples, and latent budget? Can you also report end-to-end latency and training cost? These measurements would establish whether the accuracy gains justify the additional training stages and readout supervision.