- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper Log-ICoT and shows that a L-layer transformer can provabley internalize the CoT of a 2^L parity tree. The experiment on balanced ListOps and four-digit multiplication with pretrained GPT-2 shows that Log-ICoT attains high answer accuracy with fewer training updates than vanilla ICoT and lower inference latency than explicit CoT.
Critical strengths
Provably shows that a simplified L-layer transformer trained with Log-ICoT learns k-parity, where k = 2^L, using a number of samples polynomial in the input length n and L = log2 k training stages. The experiment on n balanced ListOps and four-digit multiplication further support such claim.
Critical weaknesses
Generalization remains a limit. Experiment only limit to toy dataset rather than general dataset. Also,the method requires the balanced tree, making it hard to generalize to CoT with other structure
Questions for authors
- The idea of the Log-ICoT is interesting, but the method is strictly limit to CoT with a balanced tree. I suggest the authors to try the method with other fixed but unbalanced tree.
- Could authors futher compare the method on 7 × 7 mult. and 9 × 9 mult., or a deeper tree?
3 comments
Thank you for your response. The experiment on unbalanced trees partially addresses my concern. This is an interesting work interesting and I will maintain my original score.
Thank you for your constructive feedback and positive assessment of our work. Our general response presents additional evidence on ListOps with depth and discusses the remaining difficulty with full internalization on 8×8 multiplication. We will continue experimenting to investigate this difficulty. We appreciate your suggestions, which helped us strengthen the experimental evaluation.
Response to Reviewer gvvC — Part 1/1
Thank you for suggesting broader task structures and larger-scale experiments.
W1. Limited datasets and balanced-tree restriction
We have expanded the experiments to unbalanced trees and graph-based logical reasoning. On a fixed unbalanced ListOps tree with 16 leaves and depth 6, Log-ICoT achieves 98.0% test accuracy in 48,250 updates, compared with 97.9% in 66,250 updates for token-wise ICoT. This shows that the curriculum can work beyond balanced trees; our response to Q1 describes the setup and how the benefit depends on the amount of parallel computation. Experiments on deeper balanced trees and larger multiplication are still running, as discussed in Q2.
We also added ProsQA, a synthetic logical reasoning benchmark in which natural-language statements describe a directed acyclic graph and the model must find a valid reasoning path through distracting branches (Hao et al., 2024). We provide breadth-first search traces listing all newly reachable nodes at each step and progressively internalize these steps. After full internalization, Log-ICoT achieves 96.2% test accuracy in 22,750 updates, close to explicit CoT's 96.6%. The consolidated table in our shared response reports the results. These experiments broaden the structures considered, while still using synthetic tasks and supplied decompositions. Learning the decomposition and evaluating on broader natural-language reasoning tasks remain future work.
Q1. Fixed unbalanced trees
Following the suggestion, we evaluated a fixed unbalanced ListOps tree with 16 leaves, depth 6, and level widths 6,4,2,1,1,1. We group computations by node height so that independent nodes can be internalized together. Both methods use GPT-2 Small, 10,000 explicit-CoT updates, a learning rate of 10−4, and the same validation-based stopping rule.
From the consolidated table in our shared response, Log-ICoT uses about 27% fewer updates. Its smaller advantage than on balanced D=5 ListOps is consistent with the reduced parallelism: several levels contain only one node, leaving fewer computations to internalize together. In the limiting case of a chain, Log-ICoT removes one reasoning step at a time, matching the token-wise removal order when each step is one token. Greater parallelism allows Log-ICoT to internalize more computations together and reduce the number of curriculum stages.
Q2. Larger multiplication or deeper trees
We have extended Log-ICoT to an unbalanced ListOps tree with depth 6, deeper than the original balanced tree settings, and to ProsQA, a more complex graph-based logical reasoning task expressed in natural language.
Experiments on 8×8 multiplication and balanced ListOps D=6 are still running. We will update the consolidated table with their final test accuracies and training costs once available.
References
Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. (2024; COLM 2025). Training Large Language Models to Reason in a Continuous Latent Space. arXiv:2412.06769.