- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce LatentGrid as a means to study how continuous states are represented and how they are supervised in latent reasoning. They compare projected hidden states against top-k mixtures of vocabulary embeddings across different objectives (indirect and direct supervision, influencing different forms of optimization pressure). Their analysis shows that curriculum and reconstruction supervision improve performance, and direct supervision (reconstruction, compression) relatively preserves reasoning content while mixtures of vocabulary embeddings degrade it. The authors then propose Adaptive Curriculum Thinking (ACT) which involves progressive internalization with a direct supervision objective. Experimental validation is performed primarily with Llama-3.2-1B, and the reconstruction objective version of ACT seems to generally perform the best overall.
Critical strengths
-
The comparison of different objectives under controlled conditions is quite interesting, and the finding that unconstrained representations are generally better than constrained ones is valuable for designing future latent reasoning mechanisms.
-
The analysis in Section 4 actually motivates the design of a performant training mechanism in ACT, which seems to be generally effective across benchmarks and across both Llama- and Qwen-family models.
Critical weaknesses
-
Relevant baselines for ACT, such as Coconut, CODI (both of which are discussed in related work), and iCoT (which involves progressive internalization), are missing; overall, no latent reasoning baselines from the literature have been evaluated against.
-
The mechanistic interpretation of the probes seems a bit too strong, as recovering a rationale with a probe doesn’t necessarily suggest that the reasoner actually uses it.
Questions for authors
-
Given that ACT proposes an instantiation with token-level grounding, which is quite granular, do you think that a fixed natural language token-to-latent state mapping is actually required? Given that works such as Coconut (with curriculum but a fixed number of latents) and Abstract-CoT (separate discrete vocabulary/codebook learned without compression) do not have such a mapping, how does this relate to performance-cost trade-offs?
-
ACT adds several components at once, where the token grounding in the reconstruction setting introduces a 2 rationale tokens -> one state grouping, but we cannot yet determine whether the performance improvements should be attributed to token grounding or to adaptive stopping. Ablating these individually would be helpful to show ACT as a well-informed mechanism.
-
A more general comment on presentation: while I appreciate the built-up analysis that motivates ACT, the paper’s pacing results in the ACT formulation and results feeling quite rushed, along with details being pushed to the appendix.
-
It would be good to qualify that since ACT is compressing rationales based on length (with a mapping), it is not necessarily learning difficulty-aware reasoning, other than preserving the property that long verbalized CoTs will be proportionately long in the mapped space.
6 comments
Response to Reviewer FZ44 (Part II)
2. What the probes establish
The mechanistic interpretation of the probes seems a bit too strong, as recovering a rationale with a probe doesn’t necessarily suggest that the reasoner actually uses it.
We agree that recovering a rationale with an auxiliary probe only shows that reasoning information is decodable from the latent states, not that the reasoner actually uses it when generating the answer. We have revised Section 4.3 to clarify that probe metrics measure retained rationale information.
To examine whether the models use their latent trajectories during answer generation, we conducted two trajectory-replacement interventions on ACT-Rec and ACT-Comp. For each model, we selected 100 GSM8K test questions that it answered correctly and for which changing one numerical quantity yielded a counterfactual with a recomputed reference answer. Holding the original question fixed, we replaced the model's original latent trajectory in two ways:
- Counterfactual replacement: substituting the latent trajectory generated by the same model for the modified question.
- Random mismatch: reassigning the 100 original trajectories across the 100 questions so that no question received its own trajectory.
Over these 100 samples, we report the answer change rate (how often the prediction differs from the original correct answer) and the source-answer match rate (how often it matches the ground-truth answer of the question supplying the replacement trajectory).
| Model | Intervention | Answer change rate (%) | Source-answer match rate (%) |
|---|---|---|---|
| ACT-Rec | Counterfactual replacement | 13 | 6 |
| ACT-Rec | Random mismatch | 85 | 4 |
| ACT-Comp | Counterfactual replacement | 99 | 81 |
| ACT-Comp | Random mismatch | 100 | 94 |
The two models respond differently to these interventions. ACT-Comp follows the replacement trajectory closely: its prediction changes in 99% of counterfactual replacements and 100% of random mismatches, matching the source question's answer 81% and 94% of the time, respectively. ACT-Rec shows a different pattern. Under counterfactual replacement, only 13% of answers change and 6% match the counterfactual answer. Under random mismatch, however, 85% of answers change and only 4% match the mismatched source answer.
This difference may be related to the training objectives. ACT-Comp directly matches latent states to pooled embeddings of rationale tokens, providing concrete representation targets. ACT-Rec instead trains an auxiliary decoder to reconstruct rationale content from the latent states, leaving more flexibility in how information is encoded. As a result, when the modified question shares the same wording and the original numbers remain in the prompt, ACT-Rec's answer generator may draw contextual or structural cues from the latents while still reading specific numbers from the original question.
For both methods, random mismatch substantially reduces accuracy on the original questions, from 100% to 15% for ACT-Rec and to 0% for ACT-Comp. These results suggest that question-relevant latent reasoning plays an important role in correct answer generation in both models, although the extent to which a replacement trajectory transfers its source answer differs.
Response to Reviewer FZ44 (Part III)
3. Token-to-latent correspondence and the performance-cost trade-off
Given that ACT proposes an instantiation with token-level grounding, which is quite granular, do you think that a fixed natural language token-to-latent state mapping is actually required? Given that works such as Coconut (with curriculum but a fixed number of latents) and Abstract-CoT (separate discrete vocabulary/codebook learned without compression) do not have such a mapping, how does this relate to performance-cost trade-offs?
For our solution, a fixed natural-language token-to-latent state mapping is not required. In ACT, associating latent states with groups of explicit rationale tokens provides a clearer intermediate signal about the information to retain, while keeping the latent representations continuous. Our results indicate that this form of supervision is helpful within the evaluated ACT configurations.
In addition, we would like to clarify that token-level grounding does not assign a single fixed text token as the prediction target for each latent state. Each state is associated with a group of up to two rationale tokens, and one token is sampled uniformly from that group as the auxiliary target on each training pass. In expectation, the resulting loss equals the average token-prediction cross-entropy over the group, meaning that grounding supervises the group's token distribution rather than a single token.
Regarding the performance-cost trade-off, the grouping ratio determines how much rationale information each latent state is trained to retain. Smaller groups provide finer-grained supervision but require more latent states to represent a given reference rationale during training. Conversely, larger groups permit shorter target trajectories and may reduce inference costs, but require each state to retain more information, potentially affecting reasoning accuracy. These considerations suggest a trade-off between supervision granularity, reasoning accuracy, and computational efficiency, although systematically evaluating different grouping ratios remains an interesting direction for future work.
Response to Reviewer FZ44 (Part I)
Thank you for your careful review and for recognizing the value of the design comparisons and their connection to ACT. Below, we report additional baseline comparisons, clarify the scope of the probe results and the token-to-latent correspondence, report additional evaluations of token grouping and adaptive stopping, and describe the revisions to Section 5.
1. Comparisons with existing methods
We agree that comparing against established latent reasoning methods helps contextualize ACT's performance. To address this, we evaluated Coconut, CODI, and iCoT across the four arithmetic benchmarks. The table below integrates these baselines into Table 3, reporting answer accuracy (%) alongside the mean number of generated latent tokens in parentheses. Coconut and CODI each use a fixed budget of 12 latent tokens, whereas iCoT internalizes reasoning to predict the final answer directly without generating intermediate latent tokens. Bold and italicized values indicate the best and second-best accuracy in each column, respectively.
† As in Table 3, these variants adjust the reconstruction target window to match the token-level grouping.
Across all four benchmarks, both ACT variants consistently outperform Coconut and iCoT. Compared with CODI, which uses a fixed 12-token budget, ACT achieves competitive overall accuracy while adapting its reasoning length across tasks. On benchmarks with shorter reference rationales such as SVAMP and MultiArith, ACT-Rec closely matches CODI (within 0.4 percentage points) while using roughly half as many latent tokens (6.0–7.2 vs. 12.0). Conversely, on GSM-Hard, where reference rationales are longer, ACT extends its trajectory to 15.1–16.8 tokens and achieves higher accuracy than CODI.
While these comparisons help contextualize ACT's empirical performance, the primary focus of our work is to understand how representation constraints and supervision strategies affect latent reasoning. LatentGrid isolates these design choices under a controlled setup to examine how they shape both task performance and internal mechanics. Guided by this analysis, we design ACT to address the key bottlenecks in transitioning from explicit text to continuous reasoning. Its competitive performance against strong baselines like CODI, together with its adaptive inference length, shows that these analytical insights translate into an effective training paradigm.