acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Latent Chain-of-Thought: Disentangling Representation and Supervision

Abstract

Latent chain-of-thought (CoT) reasoning aims to accelerate inference by replacing explicit text tokens with continuous hidden states. Existing approaches implement this concept through diverse latent representations and training objectives, but the tight coupling of these elements obscures the impact of individual design choices. To address this, we propose **LatentGrid**, a unified framework that decouples representation constraints from supervision strategies. By systematically varying representation constraints and supervision strategies under a shared experimental setup, we isolate their individual and joint effects on task accuracy, rationale recoverability, distributional alignment, and optimization dynamics. Our analysis indicates that unconstrained continuous representations paired with direct rationale-derived supervision better preserve recoverable reasoning information, while curriculum learning facilitates the transition to implicit reasoning. Conversely, restricting latent states to vocabulary embedding mixtures generally degrades performance and limits representational capacity. Inspired by these findings, we further propose **A**daptive **C**urriculum **T**hinking (ACT), a novel training paradigm that integrates curriculum learning with direct and token-level supervision. This synthesis improves reasoning accuracy and enables adaptive generation lengths, suggesting that effective latent reasoning benefits from flexible representations, explicit intermediate targets, and a progressive shift from explicit text.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Latent CoT, Implicit Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FZ44OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce LatentGrid as a means to study how continuous states are represented and how they are supervised in latent reasoning. They compare projected hidden states against top-k mixtures of vocabulary embeddings across different objectives (indirect and direct supervision, influencing different forms of optimization pressure). Their analysis shows that curriculum and reconstruction supervision improve performance, and direct supervision (reconstruction, compression) relatively preserves reasoning content while mixtures of vocabulary embeddings degrade it. The authors then propose Adaptive Curriculum Thinking (ACT) which involves progressive internalization with a direct supervision objective. Experimental validation is performed primarily with Llama-3.2-1B, and the reconstruction objective version of ACT seems to generally perform the best overall.

Critical strengths
  1. The comparison of different objectives under controlled conditions is quite interesting, and the finding that unconstrained representations are generally better than constrained ones is valuable for designing future latent reasoning mechanisms.

  2. The analysis in Section 4 actually motivates the design of a performant training mechanism in ACT, which seems to be generally effective across benchmarks and across both Llama- and Qwen-family models.

Critical weaknesses
  1. Relevant baselines for ACT, such as Coconut, CODI (both of which are discussed in related work), and iCoT (which involves progressive internalization), are missing; overall, no latent reasoning baselines from the literature have been evaluated against.

  2. The mechanistic interpretation of the probes seems a bit too strong, as recovering a rationale with a probe doesn’t necessarily suggest that the reasoner actually uses it.

Questions for authors
  1. Given that ACT proposes an instantiation with token-level grounding, which is quite granular, do you think that a fixed natural language token-to-latent state mapping is actually required? Given that works such as Coconut (with curriculum but a fixed number of latents) and Abstract-CoT (separate discrete vocabulary/codebook learned without compression) do not have such a mapping, how does this relate to performance-cost trade-offs?

  2. ACT adds several components at once, where the token grounding in the reconstruction setting introduces a 2 rationale tokens -> one state grouping, but we cannot yet determine whether the performance improvements should be attributed to token grounding or to adaptive stopping. Ablating these individually would be helpful to show ACT as a well-informed mechanism.

  3. A more general comment on presentation: while I appreciate the built-up analysis that motivates ACT, the paper’s pacing results in the ACT formulation and results feeling quite rushed, along with details being pushed to the appendix.

  4. It would be good to qualify that since ACT is compressing rationales based on length (with a mapping), it is not necessarily learning difficulty-aware reasoning, other than preserving the property that long verbalized CoTs will be proportionately long in the mapped space.

6 comments
AuthorsOpenReview ↗

Response to Reviewer FZ44 (Part I)

Thank you for your careful review and for recognizing the value of the design comparisons and their connection to ACT. Below, we report additional baseline comparisons, clarify the scope of the probe results and the token-to-latent correspondence, report additional evaluations of token grouping and adaptive stopping, and describe the revisions to Section 5.

1. Comparisons with existing methods

Relevant baselines for ACT, such as Coconut, CODI (both of which are discussed in related work), and iCoT (which involves progressive internalization), are missing; overall, no latent reasoning baselines from the literature have been evaluated against.

We agree that comparing against established latent reasoning methods helps contextualize ACT's performance. To address this, we evaluated Coconut, CODI, and iCoT across the four arithmetic benchmarks. The table below integrates these baselines into Table 3, reporting answer accuracy (%) alongside the mean number of generated latent tokens in parentheses. Coconut and CODI each use a fixed budget of 12 latent tokens, whereas iCoT internalizes reasoning to predict the final answer directly without generating intermediate latent tokens. Bold and italicized values indicate the best and second-best accuracy in each column, respectively.

Training paradigmGSM8KSVAMPGSM-HardMultiArith
Baselines
Coconut33.36 (12.00)43.20 (12.00)6.97 (12.00)63.17 (12.00)
CODI51.40 (12.00)59.30 (12.00)10.69 (12.00)93.67 (12.00)
iCoT34.80 (0.00)46.80 (0.00)7.73 (0.00)72.33 (0.00)
Direct supervision with curriculum learning
Progressive Reconstruction45.41 (12.00)57.10 (12.00)10.92 (12.00)95.00 (12.00)
Progressive Compression34.34 (12.00)41.00 (12.00)8.79 (12.00)79.00 (12.00)
Direct supervision with token-level grounding
Token-grounded Reconstruction†36.16 (12.93)49.90 (6.03)8.34 (15.43)68.50 (7.39)
Token-grounded Compression40.49 (13.03)46.40 (6.03)8.79 (15.24)78.50 (7.30)
Adaptive Curriculum Thinking
ACT-Rec†49.51 (12.77)58.90 (5.98)10.99 (15.11)93.50 (7.23)
ACT-Comp49.13 (12.91)55.90 (6.10)11.52 (16.75)93.33 (7.31)

† As in Table 3, these variants adjust the reconstruction target window to match the token-level grouping.

Across all four benchmarks, both ACT variants consistently outperform Coconut and iCoT. Compared with CODI, which uses a fixed 12-token budget, ACT achieves competitive overall accuracy while adapting its reasoning length across tasks. On benchmarks with shorter reference rationales such as SVAMP and MultiArith, ACT-Rec closely matches CODI (within 0.4 percentage points) while using roughly half as many latent tokens (6.0–7.2 vs. 12.0). Conversely, on GSM-Hard, where reference rationales are longer, ACT extends its trajectory to 15.1–16.8 tokens and achieves higher accuracy than CODI.

While these comparisons help contextualize ACT's empirical performance, the primary focus of our work is to understand how representation constraints and supervision strategies affect latent reasoning. LatentGrid isolates these design choices under a controlled setup to examine how they shape both task performance and internal mechanics. Guided by this analysis, we design ACT to address the key bottlenecks in transitioning from explicit text to continuous reasoning. Its competitive performance against strong baselines like CODI, together with its adaptive inference length, shows that these analytical insights translate into an effective training paradigm.

AuthorsOpenReview ↗

Response to Reviewer FZ44 (Part II)

2. What the probes establish

The mechanistic interpretation of the probes seems a bit too strong, as recovering a rationale with a probe doesn’t necessarily suggest that the reasoner actually uses it.

We agree that recovering a rationale with an auxiliary probe only shows that reasoning information is decodable from the latent states, not that the reasoner actually uses it when generating the answer. We have revised Section 4.3 to clarify that probe metrics measure retained rationale information.

To examine whether the models use their latent trajectories during answer generation, we conducted two trajectory-replacement interventions on ACT-Rec and ACT-Comp. For each model, we selected 100 GSM8K test questions that it answered correctly and for which changing one numerical quantity yielded a counterfactual with a recomputed reference answer. Holding the original question fixed, we replaced the model's original latent trajectory in two ways:

  • Counterfactual replacement: substituting the latent trajectory generated by the same model for the modified question.
  • Random mismatch: reassigning the 100 original trajectories across the 100 questions so that no question received its own trajectory.

Over these 100 samples, we report the answer change rate (how often the prediction differs from the original correct answer) and the source-answer match rate (how often it matches the ground-truth answer of the question supplying the replacement trajectory).

ModelInterventionAnswer change rate (%)Source-answer match rate (%)
ACT-RecCounterfactual replacement136
ACT-RecRandom mismatch854
ACT-CompCounterfactual replacement9981
ACT-CompRandom mismatch10094

The two models respond differently to these interventions. ACT-Comp follows the replacement trajectory closely: its prediction changes in 99% of counterfactual replacements and 100% of random mismatches, matching the source question's answer 81% and 94% of the time, respectively. ACT-Rec shows a different pattern. Under counterfactual replacement, only 13% of answers change and 6% match the counterfactual answer. Under random mismatch, however, 85% of answers change and only 4% match the mismatched source answer.

This difference may be related to the training objectives. ACT-Comp directly matches latent states to pooled embeddings of rationale tokens, providing concrete representation targets. ACT-Rec instead trains an auxiliary decoder to reconstruct rationale content from the latent states, leaving more flexibility in how information is encoded. As a result, when the modified question shares the same wording and the original numbers remain in the prompt, ACT-Rec's answer generator may draw contextual or structural cues from the latents while still reading specific numbers from the original question.

For both methods, random mismatch substantially reduces accuracy on the original questions, from 100% to 15% for ACT-Rec and to 0% for ACT-Comp. These results suggest that question-relevant latent reasoning plays an important role in correct answer generation in both models, although the extent to which a replacement trajectory transfers its source answer differs.

AuthorsOpenReview ↗

Response to Reviewer FZ44 (Part III)

3. Token-to-latent correspondence and the performance-cost trade-off

Given that ACT proposes an instantiation with token-level grounding, which is quite granular, do you think that a fixed natural language token-to-latent state mapping is actually required? Given that works such as Coconut (with curriculum but a fixed number of latents) and Abstract-CoT (separate discrete vocabulary/codebook learned without compression) do not have such a mapping, how does this relate to performance-cost trade-offs?

For our solution, a fixed natural-language token-to-latent state mapping is not required. In ACT, associating latent states with groups of explicit rationale tokens provides a clearer intermediate signal about the information to retain, while keeping the latent representations continuous. Our results indicate that this form of supervision is helpful within the evaluated ACT configurations.

In addition, we would like to clarify that token-level grounding does not assign a single fixed text token as the prediction target for each latent state. Each state is associated with a group of up to two rationale tokens, and one token is sampled uniformly from that group as the auxiliary target on each training pass. In expectation, the resulting loss equals the average token-prediction cross-entropy over the group, meaning that grounding supervises the group's token distribution rather than a single token.

Regarding the performance-cost trade-off, the grouping ratio determines how much rationale information each latent state is trained to retain. Smaller groups provide finer-grained supervision but require more latent states to represent a given reference rationale during training. Conversely, larger groups permit shorter target trajectories and may reduce inference costs, but require each state to retain more information, potentially affecting reasoning accuracy. These considerations suggest a trade-off between supervision granularity, reasoning accuracy, and computational efficiency, although systematically evaluating different grouping ratios remains an interesting direction for future work.

Reviewer_hzmZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LatentGrid, an empirical framework for comparing latent chain-of-thought along representation and supervision axes. It implements unconstrained projected states and vocabulary-embedding mixtures, paired with answer-only, curriculum, rationale-reconstruction, or rationale-compression supervision. Seven basic latent variants are evaluated using a shared initialization and training protocol on four arithmetic benchmarks, with additional analyses of rationale recoverability, embedding-distribution alignment, and latent-position gradients. Motivated by these findings, the paper proposes Adaptive Curriculum Thinking (ACT), combining progressive replacement of explicit reasoning, direct rationale-derived targets, token-level supervision, and learned termination. ACT-Rec and ACT-Comp improve over the basic latent variants, although they remain below explicit CoT on the primary arithmetic evaluations. Supplementary experiments use Qwen3-0.6B and the non-arithmetic ProsQA benchmark, where both ACT variants outperform the reported explicit-CoT baseline with shorter reasoning sequences. The paper also gives information-theoretic interpretations of reconstruction objectives and geometric explanations of why vocabulary-hull membership alone does not ensure distributional alignment.

Critical strengths
  1. The shared-protocol empirical comparison is useful in its own right. LatentGrid makes several commonly bundled design choices more comparable by using a common backbone initialization, datasets, and evaluation procedure. The seven basic variants, four arithmetic benchmarks, and supplementary Qwen3 results provide a substantive empirical study rather than merely a taxonomy. The ProsQA evaluation also supplies evidence beyond arithmetic, although it tests ACT rather than the full grid. This controlled comparative perspective can support acceptance independently of whether ACT introduces a wholly new architecture.

  2. The complementary diagnostics and intermediate training comparisons provide informative evidence. The paper evaluates rationale recoverability using a common probe protocol, quantifies embedding-distribution differences with MMD, and measures latent-position gradients. Its recovery metrics also show that more recoverable explicit reasoning does not uniformly imply greater answer accuracy, an important qualification for interpreting latent probes. Table 3 compares progressive direct supervision, token-grounded supervision, and their combination, showing substantial improvements over basic latent training; for example, progressive reconstruction increases GSM8K accuracy from 34.50% to 45.41%, and ACT-Rec reaches 49.51%. Although these are not fully isolated effects, they demonstrate that the training choices have meaningful practical consequences.

Critical weaknesses
  1. Design effects are not fully isolated. Table 3 provides useful comparisons, but token grounding is coupled with adaptive stopping; ACT-Rec also changes reconstruction grouping, and inference budgets differ. Representation comparisons additionally change parameterization and compression losses. These confounds weaken attribution to individual design choices, the central goal of LatentGrid.

  2. The incremental contribution needs clearer positioning. The already-cited study [1] examines supervision, reconstruction/compression, gradients, and latent probes; [2–3] address latent construction/curriculum and adaptive stopping. Although their mechanisms differ, the paper does not sufficiently clarify which findings and ACT components constitute a significant advance beyond this literature.

Questions for authors
  1. Clarification and controlled evidence for the contribution of ACT: With reconstruction grouping held fixed, what evidence separates token-level grounding from adaptive stopping? In particular, can the same token-grounded model be evaluated under fixed budgets and learned termination, alongside budget-matched precursor variants? Please also clarify how much of the ACT-Rec comparison can be attributed to its changed reconstruction grouping. The existing Table 3 is useful; this question concerns the effects it cannot currently distinguish, not a claim that there are no ablations.

  2. Clarification of representation and mechanism claims: How do the authors distinguish vocabulary-constraint effects from the learned-projection and MSE/KL differences? Do available controls or sensitivity results show that the observed collapse and performance ranking are robust to these choices? What evidence, if any, shows that recoverable rationale information is used by the answer generator, or that adaptive lengths track reasoning difficulty after accounting for reference length? If these stronger interpretations are not tested, please state the conclusions as implementation-specific observations rather than established general mechanisms. New large-scale experiments are a future-work suggestion, not a prerequisite for clarifying the scope of the existing evidence.

  3. Clarification of the incremental contribution relative to specific prior work: Which findings and components are new relative to What Makes Effective Supervision in Latent Chain-of-Thought: An Information-Theoretic Analysis (arXiv:2606.20075), which is already cited? Please also position ACT against Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens (arXiv:2602.10229) and Learning When to Stop: Adaptive Latent Reasoning via Reinforcement Learning (arXiv:2511.21581). A comparison of learning targets, latent construction, termination training, and empirical conclusions would materially affect my assessment. I am not assuming the methods are equivalent or requiring all three to be reimplemented as experimental baselines.

9 comments
AuthorsOpenReview ↗

Response to Reviewer hzmZ (Part I)

We thank the reviewer for the constructive feedback and helpful suggestions. Below, we provide additional experimental evaluations, clarify what these comparisons support, and explain how LatentGrid and ACT relate to prior work.

1. Interaction between design choices

Design effects are not fully isolated. Table 3 provides useful comparisons, but token grounding is coupled with adaptive stopping; ACT-Rec also changes reconstruction grouping, and inference budgets differ. Representation comparisons additionally change parameterization and compression losses. These confounds weaken attribution to individual design choices, the central goal of LatentGrid.

We agree that Table 3 introduces token-level grounding alongside adaptive stopping, and also changes the rationale grouping for Reconstruction. To separate the effects of these choices, we conducted two additional sets of experiments.

First, we evaluated the trained checkpoints of all four token-grounded variants under a fixed inference budget of 12 latent tokens, matching the budget of the fixed-length baselines. The table below compares fixed-length inference with adaptive stopping, reporting answer accuracy (%) and the mean number of latent tokens in parentheses.

Training paradigmInferenceGSM8KSVAMPGSM-HardMultiArith
Direct supervision with token-level grounding
Token-grounded ReconstructionAdaptive36.16 (12.93)49.90 (6.03)8.34 (15.43)68.50 (7.39)
Token-grounded ReconstructionFixed 1232.30 (12.00)24.70 (12.00)7.05 (12.00)57.00 (12.00)
Token-grounded CompressionAdaptive40.49 (13.03)46.40 (6.03)8.79 (15.24)78.50 (7.30)
Token-grounded CompressionFixed 1227.75 (12.00)22.90 (12.00)6.29 (12.00)47.33 (12.00)
Adaptive Curriculum Thinking
ACT-RecAdaptive49.51 (12.77)58.90 (5.98)10.99 (15.11)93.50 (7.23)
ACT-RecFixed 1238.59 (12.00)31.50 (12.00)8.11 (12.00)70.50 (12.00)
ACT-CompAdaptive49.13 (12.91)55.90 (6.10)11.52 (16.75)93.33 (7.31)
ACT-CompFixed 1233.89 (12.00)29.60 (12.00)7.88 (12.00)67.50 (12.00)

Across all four model variants and benchmarks, fixed-length inference with 12 latent tokens yields lower accuracy than adaptive stopping. For instance, ACT-Rec drops from 49.51% to 38.59% on GSM8K, and from 58.90% to 31.50% on SVAMP. Notably, on SVAMP, the adaptive models generate about 6 latent tokens on average; forcing them to continue to 12 tokens hurts performance, indicating that extra latent steps are not helpful once the model has learned when to stop.

Second, to isolate the effect of changing how rationale tokens are grouped in Reconstruction, we trained Reconstruction variants using the same token grouping as ACT-Rec (one latent state per group of up to two consecutive rationale tokens), without token-level grounding or adaptive stopping. All variants below use unconstrained representations and generate a fixed 12 latent tokens at inference.

VariantGSM8KSVAMPGSM-HardMultiArith
Reconstruction U (Table 1)34.5045.308.1966.83
Reconstruction with token grouping28.1328.106.9051.83
Progressive Reconstruction (Table 3)45.4157.1010.9295.00
Progressive Reconstruction with token grouping34.9525.807.4354.67

Both token-grouped variants perform worse than their corresponding original-grouping baselines on all four benchmarks. On GSM8K, accuracy decreases from 34.50% to 28.13% without curriculum learning and from 45.41% to 34.95% with curriculum learning. These results suggest that changing the rationale-to-latent assignment alone does not explain the improvements in Table 3.

A mismatch between training and inference may contribute to the observed performance degradation. During training, the token-grouped models process latent chains whose lengths vary with the reference rationale, whereas at inference they must generate a fixed budget of 12 latent tokens. A fixed budget may stop reasoning too early or force unnecessary extra steps. The same length mismatch is present in the Compression paradigm, where the number of training targets also depends on the grouped rationale tokens, but inference uses a fixed budget.

AuthorsOpenReview ↗

Response to Reviewer hzmZ (Part II)

Token-level grounding helps alleviate this mismatch by supervising both intermediate content and the <end_latent> termination signal. Similar to explicit CoT, the model can continue reasoning and then switch to answer generation when it predicts the end of the reasoning process. This allows it to decide when to stop for each question, providing greater flexibility in reasoning length. Taken together, the grouping comparison and fixed-length evaluations highlight the importance of adaptive stopping in token-grounded latent reasoning. The results suggest that learned termination helps align inference with the variable-length trajectories used during training, while the contribution of token-level grounding may depend on the underlying supervision paradigm.

2. Incremental contribution

The incremental contribution needs clearer positioning. The already-cited study [1] examines supervision, reconstruction/compression, gradients, and latent probes; [2–3] address latent construction/curriculum and adaptive stopping. Although their mechanisms differ, the paper does not sufficiently clarify which findings and ACT components constitute a significant advance beyond this literature.

We appreciate this feedback. Our work makes two complementary contributions: LatentGrid systematically investigates how representation constraints and supervision strategies interact in latent reasoning, while ACT translates these diagnostic insights into a unified training paradigm that combines progressive curriculum learning, direct rationale supervision, and adaptive stopping. We will further clarify these contributions and their distinctions from prior work in the Introduction, Related Work, and Section 5.

Our primary conceptual and empirical contribution in LatentGrid is a systematic comparison of representation constraints and supervision strategies under a shared experimental protocol. While prior work such as Chen et al. primarily investigates supervision objectives and their effects on latent reasoning, LatentGrid additionally examines how unconstrained and vocabulary-constrained representations interact with indirect and direct supervision strategies. This analysis reveals previously underexplored interactions between representation design and supervision. Most notably, we show that restricting latent states to the convex hull of vocabulary embeddings does not guarantee distributional alignment and generally degrades rationale recoverability and downstream accuracy under the evaluated configurations. For example, constrained Curriculum achieves lower GSM8K accuracy (30.63% vs. 35.03%) and worse probe NLL (0.768 vs. 0.395) than its unconstrained counterpart (Tables 1 and 2).

Building on these diagnostic findings, we develop ACT to address two challenges in latent reasoning: the optimization difficulties associated with abruptly transitioning from discrete text to continuous recurrence, and the gap between continuous latent states and the pretrained vocabulary. ACT integrates direct rationale-derived supervision with progressive curriculum learning, enabling a gradual transition to continuous reasoning while encouraging the preservation of intermediate reasoning information. It further introduces token-level grounding, which supervises vocabulary predictions from grouped rationale tokens together with an <end_latent> termination signal. Unlike Latent Thoughts Tuning, which selectively introduces latent computation at uncertain token positions, ACT progressively internalizes explicit rationales into compact continuous trajectories. Unlike Learning When to Stop, which employs reinforcement learning to train a separate stopping mechanism, ACT learns adaptive termination directly through supervised fine-tuning. Together, these components provide a unified training paradigm that combines progressive latent reasoning, direct intermediate supervision, and adaptive stopping without requiring post-SFT reinforcement learning.

AuthorsOpenReview ↗

Response to Reviewer hzmZ (Part III)

3. Token-level grounding and adaptive stopping

Clarification and controlled evidence for the contribution of ACT: With reconstruction grouping held fixed, what evidence separates token-level grounding from adaptive stopping? In particular, can the same token-grounded model be evaluated under fixed budgets and learned termination, alongside budget-matched precursor variants? Please also clarify how much of the ACT-Rec comparison can be attributed to its changed reconstruction grouping. The existing Table 3 is useful; this question concerns the effects it cannot currently distinguish, not a claim that there are no ablations.

To separate the effects of token-level grounding, adaptive stopping, and the changed reconstruction grouping in Table 3, we conducted two additional sets of experiments.

First, we evaluated the trained checkpoints of all four token-grounded variants under a fixed inference budget of 12 latent tokens, matching the budget of the fixed-length precursor variants. The table below compares fixed-length inference with adaptive stopping, reporting answer accuracy (%) and the mean number of latent tokens in parentheses.

Training paradigmInferenceGSM8KSVAMPGSM-HardMultiArith
Direct supervision with token-level grounding
Token-grounded ReconstructionAdaptive36.16 (12.93)49.90 (6.03)8.34 (15.43)68.50 (7.39)
Token-grounded ReconstructionFixed 1232.30 (12.00)24.70 (12.00)7.05 (12.00)57.00 (12.00)
Token-grounded CompressionAdaptive40.49 (13.03)46.40 (6.03)8.79 (15.24)78.50 (7.30)
Token-grounded CompressionFixed 1227.75 (12.00)22.90 (12.00)6.29 (12.00)47.33 (12.00)
Adaptive Curriculum Thinking
ACT-RecAdaptive49.51 (12.77)58.90 (5.98)10.99 (15.11)93.50 (7.23)
ACT-RecFixed 1238.59 (12.00)31.50 (12.00)8.11 (12.00)70.50 (12.00)
ACT-CompAdaptive49.13 (12.91)55.90 (6.10)11.52 (16.75)93.33 (7.31)
ACT-CompFixed 1233.89 (12.00)29.60 (12.00)7.88 (12.00)67.50 (12.00)

Across all four variants and benchmarks, forcing fixed-length inference with 12 latent tokens lowers accuracy compared to adaptive stopping. For instance, ACT-Rec drops from 49.51% to 38.59% on GSM8K and from 58.90% to 31.50% on SVAMP, while ACT-Comp drops from 49.13% to 33.89% on GSM8K and from 55.90% to 29.60% on SVAMP. Notably, on SVAMP and MultiArith, adaptive models generate around 6–7 latent tokens on average; forcing them to continue to 12 tokens reduces accuracy, indicating that extra latent steps are unhelpful once the model has learned when to stop.

Second, to isolate the effect of changing how rationale tokens are grouped in Reconstruction, we trained Reconstruction and Progressive Reconstruction variants using the same two-token grouping as ACT-Rec (one latent state per group of up to two consecutive rationale tokens), without token-level grounding or adaptive stopping. All variants below use unconstrained representations and a fixed budget of 12 latent tokens at inference.

VariantGSM8KSVAMPGSM-HardMultiArith
Reconstruction U (Table 1)34.5045.308.1966.83
Reconstruction with token grouping28.1328.106.9051.83
Progressive Reconstruction (Table 3)45.4157.1010.9295.00
Progressive Reconstruction with token grouping34.9525.807.4354.67

Both token-grouped variants perform worse than their corresponding original-grouping baselines across all four benchmarks: GSM8K accuracy decreases from 34.50% to 28.13% without curriculum learning, and from 45.41% to 34.95% with curriculum learning. These results suggest that changing the rationale-to-latent assignment alone does not explain the improvements in Table 3.

Reviewer_NC7COpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how representation constraints and supervision affect latent chain-of-thought reasoning. LatentGrid compares seven variants under a shared experimental setup. The analysis covers answer accuracy, rationale recoverability, representation alignment, and gradient dynamics. Based on these findings, ACT combines curriculum learning, direct rationale supervision, and token-level grounding with adaptive stopping. Experiments on two small language models show improvements over the basic latent variants. ACT also achieves strong results on ProsQA, although it remains below explicit CoT on arithmetic benchmarks.

Critical strengths
  1. The study addresses a useful design question. Separating representation choices from supervision provides a clearer basis for comparing latent reasoning methods.
  2. The analysis goes beyond final accuracy. Reconstruction probes, distributional comparisons, and gradient measurements offer complementary views of what different training schemes learn.
  3. The findings lead to a useful method. ACT improves over the basic latent variants on both backbones. The ProsQA results also provide evidence beyond arithmetic reasoning.
Critical weaknesses
  1. The contribution of token-level grounding remains unclear. Table 3 introduces grounding together with adaptive stopping. For Reconstruction, it also changes how rationale tokens are assigned to latent states. These changes make it difficult to identify which component explains the improvement.
  2. The paper reports shorter trajectories but no measured inference latency. The accuracy trade-off is also substantial: ACT-Rec reaches 49.51% on GSM8K, compared with 60.50% for explicit CoT. (Tables 1 and 3)
Questions for authors
  1. How sensitive are the constrained variants to the mixture size and temperature? This would help clarify whether their poor performance reflects the representation family or the particular configuration.
  2. Does token-level grounding still improve performance when ACT uses fixed-length inference? For Reconstruction, a comparison retaining the same token grouping would help interpret this result.
3 comments
AuthorsOpenReview ↗

Response to Reviewer NC7C (Part I)

We thank the reviewer for the constructive feedback and for recognizing the value of our diagnostic framework. Below, we address your comments regarding the interaction between token-level grounding and adaptive stopping, the measured inference latency and accuracy trade-offs of ACT, and the sensitivity of constrained representations to mixture size and temperature.

1. Contribution of token-level grounding

The contribution of token-level grounding remains unclear. Table 3 introduces grounding together with adaptive stopping. For Reconstruction, it also changes how rationale tokens are assigned to latent states. These changes make it difficult to identify which component explains the improvement.

We agree that Table 3 introduces token-level grounding alongside adaptive stopping, and also changes the rationale grouping for Reconstruction. To separate the effects of these choices, we conducted two additional sets of experiments.

First, we evaluated the trained checkpoints of all four token-grounded variants under a fixed inference budget of 12 latent tokens, matching the budget of the fixed-length baselines. The table below compares fixed-length inference with adaptive stopping, reporting answer accuracy (%) and the mean number of latent tokens in parentheses.

Training paradigmInferenceGSM8KSVAMPGSM-HardMultiArith
Direct supervision with token-level grounding
Token-grounded ReconstructionAdaptive36.16 (12.93)49.90 (6.03)8.34 (15.43)68.50 (7.39)
Token-grounded ReconstructionFixed 1232.30 (12.00)24.70 (12.00)7.05 (12.00)57.00 (12.00)
Token-grounded CompressionAdaptive40.49 (13.03)46.40 (6.03)8.79 (15.24)78.50 (7.30)
Token-grounded CompressionFixed 1227.75 (12.00)22.90 (12.00)6.29 (12.00)47.33 (12.00)
Adaptive Curriculum Thinking
ACT-RecAdaptive49.51 (12.77)58.90 (5.98)10.99 (15.11)93.50 (7.23)
ACT-RecFixed 1238.59 (12.00)31.50 (12.00)8.11 (12.00)70.50 (12.00)
ACT-CompAdaptive49.13 (12.91)55.90 (6.10)11.52 (16.75)93.33 (7.31)
ACT-CompFixed 1233.89 (12.00)29.60 (12.00)7.88 (12.00)67.50 (12.00)

Across all four model variants and benchmarks, fixed-length inference with 12 latent tokens yields lower accuracy than adaptive stopping. For instance, ACT-Rec drops from 49.51% to 38.59% on GSM8K, and from 58.90% to 31.50% on SVAMP. Notably, on SVAMP, the adaptive models generate about 6 latent tokens on average; forcing them to continue to 12 tokens hurts performance, indicating that extra latent steps are not helpful once the model has learned when to stop.

Second, to isolate the effect of changing how rationale tokens are grouped in Reconstruction, we trained Reconstruction variants using the same token grouping as ACT-Rec (one latent state per group of up to two consecutive rationale tokens), without token-level grounding or adaptive stopping. All variants below use unconstrained representations and generate a fixed 12 latent tokens at inference.

VariantGSM8KSVAMPGSM-HardMultiArith
Reconstruction U (Table 1)34.5045.308.1966.83
Reconstruction with token grouping28.1328.106.9051.83
Progressive Reconstruction (Table 3)45.4157.1010.9295.00
Progressive Reconstruction with token grouping34.9525.807.4354.67

Both token-grouped variants perform worse than their corresponding original-grouping baselines on all four benchmarks. On GSM8K, accuracy decreases from 34.50% to 28.13% without curriculum learning and from 45.41% to 34.95% with curriculum learning. These results suggest that changing the rationale-to-latent assignment alone does not explain the improvements in Table 3.

A mismatch between training and inference may contribute to the observed performance degradation. During training, the token-grouped models process latent chains whose lengths vary with the reference rationale, whereas at inference they must generate a fixed budget of 12 latent tokens. A fixed budget may stop reasoning too early or force unnecessary extra steps. The same length mismatch is present in the Compression paradigm, where the number of training targets also depends on the grouped rationale tokens, but inference uses a fixed budget.

AuthorsOpenReview ↗

Response to Reviewer NC7C (Part II)

Token-level grounding helps alleviate this mismatch by supervising both intermediate content and the <end_latent> termination signal. Similar to explicit CoT, the model can continue reasoning and then switch to answer generation when it predicts the end of the reasoning process. This allows it to decide when to stop for each question, providing greater flexibility in reasoning length. Taken together, the grouping comparison and fixed-length evaluations highlight the importance of adaptive stopping in token-grounded latent reasoning. The results suggest that learned termination helps align inference with the variable-length trajectories used during training, while the contribution of token-level grounding may depend on the underlying supervision paradigm.

2. Measured inference latency and the accuracy trade-off

The paper reports shorter trajectories but no measured inference latency. The accuracy trade-off is also substantial: ACT-Rec reaches 49.51% on GSM8K, compared with 60.50% for explicit CoT. (Tables 1 and 3)

To directly evaluate inference speed, we measured the average inference time per query for ACT-Rec, ACT-Comp, and explicit CoT across the full test sets on a single NVIDIA A40 GPU (batch size 64, BF16, greedy decoding).

ModelGSM8KSVAMPGSM-HardMultiArith
Explicit CoT33.2218.2741.3514.82
ACT-Rec21.8910.8825.259.87
ACT-Comp21.9111.5532.959.61

Values are milliseconds per query; lower is better. ACT uses adaptive stopping in these measurements.

Both ACT variants reduce inference time across all benchmarks. Relative to explicit CoT, ACT-Rec achieves a 1.50–1.68× speedup (a 33.4–40.4% time reduction) and ACT-Comp achieves a 1.25–1.58× speedup (a 20.3–36.8% time reduction). These measurements confirm that shorter latent trajectories translate into clear wall-clock efficiency gains.

At the same time, the accuracy trade-off is kept within a reasonable range. Compared with the basic latent variants on GSM8K (31.69%–35.03%), ACT-Rec improves accuracy to 49.51%, substantially narrowing the gap to explicit CoT while delivering a 1.52× speedup. On MultiArith, ACT-Rec (93.50%) and ACT-Comp (93.33%) closely approach explicit CoT (96.83%), trading only approximately 3.4 percentage points of accuracy for a 1.5× speedup.

More broadly, implicit reasoning aims to shorten reasoning trajectories and reduce inference cost while preserving as much of explicit CoT's accuracy as possible. Despite progress in recent approaches such as Coconut, CODI, and SIM-CoT, noticeable accuracy gaps persist in many settings, particularly on challenging arithmetic reasoning tasks. Narrowing these gaps while retaining inference efficiency remains an important direction for future work.

3. Sensitivity to mixture size and temperature

How sensitive are the constrained variants to the mixture size and temperature? This would help clarify whether their poor performance reflects the representation family or the particular configuration.

To examine sensitivity to the mixture size (kk) and temperature (TT), we evaluated constrained Reconstruction with five configurations: (k,T)∈{(10,0.8),(10,1.0),(10,1.5),(20,1.0),(30,1.0)}(k, T) \in \{(10, 0.8), (10, 1.0), (10, 1.5), (20, 1.0), (30, 1.0)\}. We tracked the latent representations over the first 1,000 optimizer updates on 64 validation examples. The table below reports the effective rank of the latent representations across these examples, where step 0 is before the first optimizer update.

Optimizer updatesk=10, T=0.8k=10, T=1.0k=10, T=1.5k=20, T=1.0k=30, T=1.0
08.158.469.708.468.46
30011.4810.377.0510.643.51
50014.577.471.111.721.83
7004.394.121.111.601.78
1,0002.341.601.771.961.90

Across the five configurations, the effective rank drops substantially within 1,000 updates, falling from initial values of 8.15–9.70 to 1.60–2.34. This reduction indicates a tendency toward representation collapse, where latent states lose diversity across examples. Varying kk and TT alters the rate of this decline during early updates, but does not prevent the eventual collapse. These findings suggest that the observed collapse is not specific to the default mixture size or temperature.

We also found that individual latent representations become dominated by a single vocabulary token during training. The next table reports the mean probability of the top-1 token across the validation examples.

AuthorsOpenReview ↗

Response to Reviewer NC7C (Part III)

Optimizer updatesk=10, T=0.8k=10, T=1.0k=10, T=1.5k=20, T=1.0k=30, T=1.0
00.80630.74510.59810.72680.7168
3000.55180.45490.62370.34420.8791
5000.35230.57720.98250.99520.9759
7000.48700.77100.98280.87780.9774
1,0000.88490.96950.96750.99970.9993

By step 1,000, the mean top-1 token probability reaches 0.9675–0.9997 for (10,1.0)(10, 1.0), (10,1.5)(10, 1.5), (20,1.0)(20, 1.0), and (30,1.0)(30, 1.0), and increases to 0.8849 for (10,0.8)(10, 0.8). As this probability approaches 1, the latent representation approaches a single token embedding, reducing its ability to encode richer information through a mixture of token embeddings.

4. Token-level grounding under fixed-length ACT inference

Does token-level grounding still improve performance when ACT uses fixed-length inference? For Reconstruction, a comparison retaining the same token grouping would help interpret this result.

Please refer to the fixed-length and token-grouping comparisons in our response to Point 1. Forcing ACT to generate 12 latent tokens reduces accuracy relative to adaptive inference, while adopting token grouping alone also lowers Reconstruction accuracy relative to the original grouping. These results suggest that token-level grounding and adaptive stopping are closely coupled. The learned termination signal lets the model decide when to stop reasoning and generate an answer, helping align inference with the variable-length trajectories used during training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.