- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper tests whether LLMs can silently commit to a specific spatial arrangement of objects without stating it out loud. Across 13 models, the authors evaluate thought-privacy, within-branch self-consistency, and cross-order agreement. They found that models struggle to maintain a stable silent state: reasoning models often leak the arrangement, and non-leaking models become inconsistent as the number of objects grows. Probing reveals that internal representations fail to reliably track earlier relations. Finally, the authors conclude that LLMs require explicit in-context text to remain relationally consistent.
Critical strengths
- The paper asks a well-motivated question: can LLMs commit to an unspoken relational arrangement? It splits this into thought-privacy, self-consistency, and cross-order agreement. The split helps because within-branch consistency can be achieved without any real commitment, while cross-order agreement cannot.
- The paper evaluated 13 models. The results provide initial evidence that models struggle to keep a fixed arrangement without verbalizing it. This matters for CoT monitoring and embodied agents.
- Branching identical saved prefixes into differently ordered query branches is a neat way to test commitment. Under greedy decoding, every branch sees exactly the same state at READY. So, disagreement across orders really mean the model never fixed an arrangement, rather than reflecting sampling noise.
Critical weaknesses
- The evaluation design has some key weaknesses:
- CoT disclosure is a questionable stand-in for “speaking.” Most models are not naturally trained to keep the reasoning “silent”. Penalizing CoT disclosure may simply test the models’ ability of instruction-following instead.
- After thought-privacy screening, several models have very tiny samples left, e.g. 10 for GLM-5.3, zero for GLM-5.2. The measurement thus becomes unmeasurable at this point. Conclusions are made based on insufficient results, and even available results (Figure 3) are very noisy.
- The ablation study results are built in the setup:
The paper tries to separate copying (the replayed text contains the arrangement itself, so the model reads it back) from scaffolding, where the replayed text contains cues or intermediate reasoning that help the model rebuild or stabilize an arrangement. However, it does not split the two clearly with the experiment setup. It’s possible that the comparison was between having the answer key in front and having it removed. Anyone would score better with the answer key.
Questions for authors
- Have you tried replaying the reasoning text with the arrangement statements redacted?
- Can you report confidence intervals for Figure 3, and the number of retained trajectories per point?
- The AI use statement notes that AI assistants were used to clean and reformat experimental data and support analysis. Please clarify whether these steps were performed through deterministic scripts or interactively. Will the authors release the full generation and scoring code, the raw transcripts and judge verdicts for all models, and the probe weights and hidden-state caches?
3 comments
Thank you for your response. My concerns have been addressed.
Thank you very much for your valuable feedback and support! We are glad that our responses have addressed your concerns, and we appreciate your time and effort in helping us improve our paper.
Thank you for the careful assessment and for highlighting the well-motivated question, 13-model evaluation, and neat way to test commitment.
1. Does disclosure mainly measure instruction following? (W1).
We agree that most models are not naturally trained to keep the reasoning "silent". But testing CoT disclosure is not simply testing the models' ability of instruction-following.
Our evaluation = thought-privacy × self-consistency. The model has the ability to follow the two instructions separately, but it's hard for the model to do both. That reflects LLM's limitations, as humans have the ability to think in a self-consistent way without exposing our CoT. This evaluation helps to understand the boundaries of model capabilities and the differences between human and model intelligence, and can be used to test models once they have such ability.
2. Confidence intervals, sample numbers, noisy result (W2&Q2).
Thank you for your careful review. We've added standard deviation intervals and sample numbers in the updated manuscript. For the noisy result, we identify the root cause, rerun the whole experiment and update Figure 3 using new results. (Please refer to response "2. Could a fixed direction label explain the high scores?" and "3. Are larger-model results supported by enough samples? Provide retained samples, and uncertainty. " of Reviewer#ah54)
3. Does replay help because the answers are already in the text? (W3&Q1)
We want to clarify that the ablation study mainly wants to claim that "returning previous CoT is important for maintaining self-consistency and agreement"(see Section 3.6.). This has practical value because some ChatCompletionAPI of models like DeepSeek/GLM/GPT don't return previous CoT by default.
We agree that there may contain both answers that can be directly-copied and cues or intermediate reasoning. It's an interesting question, and we follow your suggestions to conduct a redaction experiment, preserving questions and previous final responses while deleting CoT lines leaking adopted arrangement relations beyond their transitive closure. We use MiMo-V2.6-Pro and GPT-6.1-Sol to do the redaction, and compare the results of "full CoT" with "the redacted ones":
These results demonstrate answer sensitivity to the redaction intervention. We believe this provides a more fine-grained attribution of changes in results, thank you for your suggestion.
4. Deterministic VS Interactively AI-assisted processing? Open-source? (Q3).
Yes, we will open-source our code and data, including all the above things you mentioned. Please refer to this anonymous link for preview: https://anonymous.4open.science/r/Keep-In-Mind-LLM-77B5/ (P.S. hidden-state caches are too large that exceeds the 2GB size limit of anoynomous github, so we provide the code to generate hidden-state caches instead.)
Please let us know whether the redaction results and the clarifications on sample support and reproducibility address your concerns, or whether any points remain unclear :)