acceptodds
Under review as a conference paper at ICLR 2027

Keep in mind: Can LLMs imagine object relations without speaking?

Abstract

Humans can imagine relations among objects without verbalizing them, yet visual working memory holds only a handful of objects—about three to four—without verbal support. It remains poorly understood whether large language models (LLMs) possess this ability and what its limits are. The question matters for downstream applications: reliable spatial reasoning about object layouts is essential for embodied AI, and whether a model can reason in its hidden space affects the effectiveness of chain-of-thought (CoT) monitoring in AI safety. To characterize this capability, we introduce a simple evaluation setting that separates three requirements: thought-privacy, self-consistency, and cross-order agreement. We evaluate 13 models spanning a wide size range (1B to 744B parameters), from standard transformers to looped transformers, with and without CoT. Our experiments show that, for current open-source reasoning models, thought-privacy is hard to achieve regardless of model size or architecture. When thought-privacy is achieved, self-consistency and cross-order agreement are hard to attain. These results indicate that explicit text continues to play an important role in sustaining relational consistency. To examine the internal activity accompanying these answers, we extend the Jacobian lens (J-lens) to measure how hidden states align with groups of object names and direction words in Qwen3.8-27B. Together with the relation-readout and reasoning-text replay results, these findings suggest that relational consistency across questions relies on explicit records of prior reasoning in the context rather than on a stable, unspoken private state. Overall, models struggle to maintain a fixed arrangement of object relations without verbalizing it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
relational consistency, spacial reasoning, large language models, concealment, looped transformers, J-lens, chain-of-thought

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TLgNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper tests whether LLMs can silently commit to a specific spatial arrangement of objects without stating it out loud. Across 13 models, the authors evaluate thought-privacy, within-branch self-consistency, and cross-order agreement. They found that models struggle to maintain a stable silent state: reasoning models often leak the arrangement, and non-leaking models become inconsistent as the number of objects grows. Probing reveals that internal representations fail to reliably track earlier relations. Finally, the authors conclude that LLMs require explicit in-context text to remain relationally consistent.

Critical strengths
  1. The paper asks a well-motivated question: can LLMs commit to an unspoken relational arrangement? It splits this into thought-privacy, self-consistency, and cross-order agreement. The split helps because within-branch consistency can be achieved without any real commitment, while cross-order agreement cannot.
  2. The paper evaluated 13 models. The results provide initial evidence that models struggle to keep a fixed arrangement without verbalizing it. This matters for CoT monitoring and embodied agents.
  3. Branching identical saved prefixes into differently ordered query branches is a neat way to test commitment. Under greedy decoding, every branch sees exactly the same state at READY. So, disagreement across orders really mean the model never fixed an arrangement, rather than reflecting sampling noise.
Critical weaknesses
  • The evaluation design has some key weaknesses:
  1. CoT disclosure is a questionable stand-in for “speaking.” Most models are not naturally trained to keep the reasoning “silent”. Penalizing CoT disclosure may simply test the models’ ability of instruction-following instead.
  2. After thought-privacy screening, several models have very tiny samples left, e.g. 10 for GLM-5.3, zero for GLM-5.2. The measurement thus becomes unmeasurable at this point. Conclusions are made based on insufficient results, and even available results (Figure 3) are very noisy.
  • The ablation study results are built in the setup:

The paper tries to separate copying (the replayed text contains the arrangement itself, so the model reads it back) from scaffolding, where the replayed text contains cues or intermediate reasoning that help the model rebuild or stabilize an arrangement. However, it does not split the two clearly with the experiment setup. It’s possible that the comparison was between having the answer key in front and having it removed. Anyone would score better with the answer key.

Questions for authors
  1. Have you tried replaying the reasoning text with the arrangement statements redacted?
  2. Can you report confidence intervals for Figure 3, and the number of retained trajectories per point?
  3. The AI use statement notes that AI assistants were used to clean and reformat experimental data and support analysis. Please clarify whether these steps were performed through deterministic scripts or interactively. Will the authors release the full generation and scoring code, the raw transcripts and judge verdicts for all models, and the probe weights and hidden-state caches?
3 comments

Thank you for the careful assessment and for highlighting the well-motivated question, 13-model evaluation, and neat way to test commitment.

1. Does disclosure mainly measure instruction following? (W1).

We agree that most models are not naturally trained to keep the reasoning "silent". But testing CoT disclosure is not simply testing the models' ability of instruction-following.

Our evaluation = thought-privacy × self-consistency. The model has the ability to follow the two instructions separately, but it's hard for the model to do both. That reflects LLM's limitations, as humans have the ability to think in a self-consistent way without exposing our CoT. This evaluation helps to understand the boundaries of model capabilities and the differences between human and model intelligence, and can be used to test models once they have such ability.

2. Confidence intervals, sample numbers, noisy result (W2&Q2).

Thank you for your careful review. We've added standard deviation intervals and sample numbers in the updated manuscript. For the noisy result, we identify the root cause, rerun the whole experiment and update Figure 3 using new results. (Please refer to response "2. Could a fixed direction label explain the high scores?" and "3. Are larger-model results supported by enough samples? Provide retained samples, and uncertainty. " of Reviewer#ah54)

3. Does replay help because the answers are already in the text? (W3&Q1)

We want to clarify that the ablation study mainly wants to claim that "returning previous CoT is important for maintaining self-consistency and agreement"(see Section 3.6.). This has practical value because some ChatCompletionAPI of models like DeepSeek/GLM/GPT don't return previous CoT by default.

We agree that there may contain both answers that can be directly-copied and cues or intermediate reasoning. It's an interesting question, and we follow your suggestions to conduct a redaction experiment, preserving questions and previous final responses while deleting CoT lines leaking adopted arrangement relations beyond their transitive closure. We use MiMo-V2.6-Pro and GPT-6.1-Sol to do the redaction, and compare the results of "full CoT" with "the redacted ones":

MetricMiMo-V2.6-ProGPT-6.1-Sol
Redacted samples207156
Relation answers changed after redaction433/999 (43.3%)325/782 (41.6%)
Agreement with known axis relations: full to redacted98.9% → 79.2%99.5% → 89.6%

These results demonstrate answer sensitivity to the redaction intervention. We believe this provides a more fine-grained attribution of changes in results, thank you for your suggestion.

4. Deterministic VS Interactively AI-assisted processing? Open-source? (Q3).

  • Deterministic: Self-consistency, cross-order agreement, aggregation, and plotting are computed by fixed scripts.
  • Interactively: Our analysis uses an LLM judge for disclosure and explicit revision, the judge prompt was refined manually over multiple iterations, but the original model outputs and recorded labels were never edited. The original paper provides the frozen V3.7 system and user prompts in Appendix E (Sections E.1–E.2).

Yes, we will open-source our code and data, including all the above things you mentioned. Please refer to this anonymous link for preview: https://anonymous.4open.science/r/Keep-In-Mind-LLM-77B5/ (P.S. hidden-state caches are too large that exceeds the 2GB size limit of anoynomous github, so we provide the code to generate hidden-state caches instead.)

Please let us know whether the redaction results and the clarifications on sample support and reproducibility address your concerns, or whether any points remain unclear :)

Thank you for your response. My concerns have been addressed.

Thank you very much for your valuable feedback and support! We are glad that our responses have addressed your concerns, and we appreciate your time and effort in helping us improve our paper.

Reviewer_XuAeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs can maintain a stable internal representation of object relations without explicitly verbalizing them. The authors propose an evaluation framework that measures three capabilities: thought-privacy, self-consistency, and cross-order agreement. They evaluate 13 models ranging from 1B to 744B parameters using tasks that require models to privately choose spatial arrangements and consistently answer pairwise relation queries presented in different orders. The results show that models struggle to maintain consistent relations without explicitly recording them, and that replaying previous reasoning text substantially improves consistency. Furthermore, internal representation analysis using the Jacobian lens and linear probes reveals that current relational judgments can be decoded from hidden states, but the decoded relations are not consistently preserved across subsequent questions. These findings suggest that explicit reasoning text plays an important role in maintaining relational consistency, highlighting limitations in LLMs' ability to sustain stable, unspoken relational representations.

Critical strengths

The paper investigates a relatively underexplored aspect of text-only LLMs: whether they can maintain stable representations of self-generated spatial relations without explicitly verbalizing them. The proposed evaluation framework provides a useful perspective for distinguishing thought-privacy, self-consistency, and cross-order agreement, and the experiments reveal interesting limitations in the relational consistency of current models.

Critical weaknesses

I'm not an expert in LLMs but work more on VLM-related topics.

In my opinion, the limitations are

  • Limited practical implications. The paper mainly characterizes a limitation of current LLMs without proposing a new algorithm or mechanism to address it. The implications of these findings for improving LLM reasoning or designing more reliable agent systems remain unclear.
  • The role of explicit memory is relatively straightforward. The finding that replaying previous reasoning text substantially improves consistency is interesting, but explicitly recording and reusing a spatial arrangement appears to be a simple and effective solution. It would be helpful to clarify what additional insights the study provides beyond demonstrating the importance of explicit state recording, and in which settings such a solution may be insufficient.
  • Limited evaluation of visually grounded spatial memory. The experiments focus on abstract text-based spatial relations. Although this setting isolates relational consistency from visual perception, it remains unclear whether the findings generalize to spatial memory grounded in images or videos, which is more directly relevant to embodied AI and real-world spatial reasoning.
Questions for authors

Refer to weaknesses.

4 comments

Thank you for recognizing the value of studying this underexplored question and separating different aspects of relational consistency. You pointed out that our original evaluation was limited to text and left open whether related difficulties occur in visually grounded spatial task. This is indeed an interesting question.

Following your suggestion, we added two visual experiments. We found that a model's own written description does not always preserve the performance as from the image, and that retained internal state can provide useful information beyond the same written record. These findings help explain both the usefulness and the limitations of explicit recording. Thank you for suggesting this extension: it helped us examine these questions in a broader visual setting.

Below, we explain the practical implications (W1), what we learn beyond the usefulness of explicit records (W2), and the visual reconstruction results (W3). Since the experiments contain many pictures, we provide an anonymous link with main results for your convenience: https://spatial-memory-evidence-276d51.vercel.app/

1. How can these findings help build more reliable systems? (W1)

You asked how our findings could inform better reasoning or memory systems. Our evaluation provides a way to check whether a model actually preserves the relations it is supposed to preserve, and we've found it is difficult to preserve consistently without speaking for all models, and larger models are not necessarily better than smaller ones.

A model may give compatible answers within one conversation but describe a different arrangement when the questions are reordered. Checking only one conversation would miss this problem. Self-consistency checks whether the answers fit a possible arrangement; cross-order agreement checks whether the same relations are preserved across question orders. Thought-privacy separately checks whether the arrangement has appeared in accessible text.

These checks can help compare prompts and memory methods and identify what each method improves. Our additional visual experiments make this practical question concrete: when a model writes a record of a scene, is that record sufficient, and does its retained internal state contribute anything beyond it? We examine these questions below.

2. Why not simply write down the arrangement? (W2)

We agree with you that accurately recording an arrangement and reading it later is a useful solution. Our paper concerns two more questions: is the generated record sufficient, and does retained internal state contribute anything beyond it? To test this, we conduct an experiment on 40 visual scenes based on the IR3D dataset, with five scenes at each object count from N=3N=3 to N=10N=10. The model is given an image showing multiple objects of different shapes on a table. Each model first describes an image. We then ask it to recover the left-to-right order of the objects under three conditions:

  • V: retain the state and allow access to image tokens.
  • H: retain the internal states of text but mask internal states of the image tokens.
  • R: rebuild state without the image, using exactly the same text tokens at the same positions. The left-to-right relation F1 score is as follows: | Model | V | H | R | H - R | |---|---:|---:|---:|---:| | Qwen3-VL-8B | 77.59 | 60.45 | 60.32 | 0.13 | | Qwen3-VL-32B | 84.21 | 65.55 | 57.74 | 7.81 | | Qwen3.8-27B | 91.55 | 87.39 | 79.88 | 7.51 | | Gemma 4 31B | 90.30 | 81.25 | 80.44 | 0.82 | | Ministral 3 14B Instruct | 87.22 | 74.80 | 70.23 | 4.57 |

R already recovers substantial information, confirming the usefulness of explicit records. However, H provides an additional mean benefit for some checkpoints, especially Qwen3-VL-32B and Qwen3.8-27B. Thus, rereading the same words does not always reproduce the performance obtained with retained state. The retained internal state has additional value.

Beyond benefits, our paper also shows the limitations of the internal memory. Through internal testing, we found that the model may processes relations and objects in different layers, and when the model processes the relationship between A and B asked by the user, the relative position relationship between A and C, D, etc. in the internal state may not be always maintained.

3. Does the problem also occur in visual settings? (W3)

Using the aforementioned 40 IR3D scenes, the model is asked to reconstruct the corresponding 3D scene using Blender. For each model and image, we generate a scene description first. Then we compare three reconstruction conditions: Text (T) reconstructs from this record after removing the image; Image (I) reconstructs directly from the original image; Image+text (IT) receives both the original image and the text. The result is as follows:

ModelValid outputs T/I/IT, out of 40Common scenesText only (T)Image only (I)Image + text (IT)
Gemini 4 Argon39/40/403989.493.190.5
GPT-6 Astra, high reasoning22/22/191874.885.980.1
Claude Opus 5.536/36/363677.080.172.3
Qwen3.6-35B-A3B39/39/393951.653.150.9

The results show that the helpfulness and limitation of text description. It is helpful for the model to better understand the 3D front-back relation in 2D image, and the location of the object; It is less helpful for understanding left and right, color and shape. With the increase of n, it is more and more difficult to understand the global relationship by text description, but it is better to understand directly based on the original picture.

Thank you again for your thoughtful review. With less than a day remaining in the discussion period, we would appreciate your thoughts on our response and whether it addresses your concerns. If any concerns remain, we would welcome the opportunity to clarify them before the deadline :)

Thanks for your reply. I would keep my scores, but since my expertise is not exactly in this domain, I'm not going to champion this paper.

Reviewer_ah54OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLMs can maintain a self-selected, unverbalized relational configuration over a set of objects. The model is instructed to internally fix an arrangement of NN objects on a line (1D) or plane (2D) and to acknowledge with READY token. The resulting conversation prefix is then branched into four continuations, which query all (N2)\binom{N}{2} pairwise relations in different orders, and a direct-reveal branch. The authors decompose the capability into three components, each compared against a random baseline: (i) thought-privacy, assessed by an LLM judge for disclosure or revision of the arrangement in visible reasoning and responses; (ii) self-consistency, defined as acyclicity of the within-branch relation graph; and (iii) cross-order agreement, the fraction of pairwise relations shared across branches.

The behavioral evaluation covers 13 open-weight and API models (1B to 744B parameters), including reasoning variants and looped/recurrent-depth architectures (Huginn, Ouro). It spans N∈{2,…,12}N \in \{2,\dots,12\}, both dimensionalities and two languages, under greedy decoding. Reasoning models frequently disclose their arrangement in their chain of thought (e.g., about 94% initial disclosure for Qwen3.8-27B in 2D). Among trajectories that pass the privacy screen, self-consistency and agreement degrade toward random chance as NN increases. A replay ablation shows that keeping prior reasoning in context substantially improves agreement (+60.4 points for Qwen3.8-27B in 2D).

The mechanistic analysis on Qwen3.8-27B has two parts. First, the authors extend the J (Jacobian) lens with nonnegative cones over object-name and direction-word directions and report distinct depth profiles, with relation alignment peaking near layer 35 and object alignment at the final layer. Second, a linear probe in a gradient-derived subspace at layer 35 decodes the current relational answer with high accuracy (87.5% in 1D, 85.1% joint in 2D). The same probe reads out previously queried relations only at chance at later questions. The authors found that the model stays consistent because it reads past text records, not because it has a continuous internal memory.

Critical strengths
  • The evaluation protocol is well designed. Branching a shared prefix into query branches with permuted orders turns the question of whether a model maintains a fixed latent choice into a measurable order specific bias test that requires no ground-truth arrangement.
  • The decomposition into thought-privacy, self-consistency and cross-order agreement, together with closed-form chance baselines, separates distinct failure modes.
  • The reporting is transparent viz. planned and eligible counts are kept separate, task and judge prompts are shared, and test-split reuse is mentioned. As a result, the behavioral findings are reproducible, and the protocol can readily be adopted for CoT-monitorability research.
Critical weaknesses

The paper's main claim about the model's internals is not supported well. The paper claims that models have no hidden internal memory of the arrangement and rely only on what they wrote. This claim rests on the probe and Jacobian-lens analyses, which have few issues:

  • The probe fails even when the information is known to be there. When the model's earlier reasoning is kept, Qwen3.8-27B answers with perfect consistency in 1D, and the arrangement is written in its context. The probe still reads earlier answers at chance (48.3% in 1D, 24.1% in 2D). A probe that misses information the model is clearly using cannot show that the information is absent.
  • The claims about model size and architecture rest on small language models. After leaks are removed, the results for three or more objects come mostly from Qwen3-4B-Instruct and from looped models with 1.4B to 3.5B parameters. Qwen3.8-27B and GLM-5.2 have no data left, and no large model is tested with reasoning turned off. The plots show neither sample sizes nor confidence intervals.
  • Random guessing seems like a weak baseline. Each question lists the alphabetically earlier object first, so a model that always gives the same answer looks perfectly consistent. That pattern breaks the instruction not to use alphabetical order, but the paper never checks for it. Ouro-1.4B's agreement in 2D rises from 30% with 6 objects to 66% with 12 objects. This fits the pattern, and the paper does not explain it.
  • An LLM judge decides which conversations leak, with no check against human labels.
Questions for authors

Q1: The probe reads earlier pairs at chance even when the arrangement is written in context and the model answers with perfect consistency, so the null result may reflect the probe rather than the model. Could you add a positive control? For example, train a readout on all pairwise relations from the direct-reveal answer, and show that it recovers earlier pairs when the arrangement is in context.

Q2: A model that always gives the same label scores perfectly, because queries list the alphabetically earlier object first. How often do the answers imply the alphabetical order or use a single label, and does this explain why Ouro-1.4B improves as NN grows?

Q3: The privacy screen relies entirely on an LLM judge. How well does the judge agree with human labels on a sample of trajectories?

3 comments

Thank you for acknowledging that the "evaluation protocol is well designed", and "decomposition separates distinct failure modes", as well as "transparent reporting" and "reproducible results".

We address probe results, label shortcuts, sample support, and judge reliability separately.

1. Could the failed readout reflect a limitation of the probe? (W1,Q1)

We sincerely thank the reviewer for this insightful comment. We agree that low readout accuracy alone cannot definitively establish that information is completely absent from the context.

However, we would like to clarify the global nature of our probing formulation and share new positive control results. Specifically, our method learns a representation vector for every object across the entire dataset; for any queried pair, we project their corresponding vectors onto a fixed axis and use the sign of their projection difference to predict their relative spatial relation (e.g., left–right). Consequently, our probe is theoretically capable of making global predictions across all object pairs at any query position.

Furthermore, we conducted a direct-reveal positive control experiment and updated to Appendix D.4 (thanks for your advice!), in which we first explicitly reveal the ordering among objects in the context before using the probe to predict their spatial relations. Under this setting, the probe achieves 93.08% in 1D and 89.62% in 2D, higher than the accuracy when the object relations are not directly revealed in the context (and in contrast to the cross-query replay scores of 48.3% in 1D and 24.1% in 2D).

TaskAccuracy
1D Left–Right Relation93.08%
2D Left–Right Relation89.62%

The probing experiments support our core claim: the model struggles to maintain cross-query consistency in the latent space.

2. Could a fixed direction label explain the high scores? (W3&Q2)

Your concern is valid. Although our instructions prohibit trivial arrangements, some original queries allow repeated direction labels to produce a self-consistent arrangement.

Reanalysis of the Ouro-1.4B results shows that NE answers rise from 44.29% at N=6N=6 to 78.03% at N=12N=12. Only 1/22 retained trajectories is strictly constant-label, so the broader issue is strong direction bias.

To fix that, we've rerun the experiments with randomized query order to reduce the effects of biased direction, and check that constant-label answers can't reach self-consistency on the whole set. Please refer to Figure 3 in the updated PDF manuscript.

3. Are larger-model results supported by enough samples? Provide retained samples, and uncertainty. (W2)

You point out that screening only leaves small models, and the conclusion may not apply for larger models. We've done the following to improve:

(1) We add an "ALL" graph to show the results before disclosure removal, which provides the bigger picture with all valid trajectories.

(2) Large models lack remaining trajectories, we therefore added more large models of 27–744B (e.g. Nanbeige4.3-33B and Llama-3.3-70B-Instruct) and found that they're better than smaller models at self-consistency and cross-order agreement when thought-privacy is not guaranteed, but behave still badly after thought-privacy filtering.

(3) For the several models that have very tiny samples left, we report the mean score and standard deviation intervals and provide the number of retained trajectories per point. (Please refer to *Figure 3 in the updated PDF, and Appendix B&C for more details).

4. Do we check the judge? (W4&Q3)

We randomly sample 30 judge results: 10 each for initial disclosure, later disclosure and revision. Two human evaluators independently checked them using definitions aligned with the judge prompt, without seeing any LLM judge labels. We also added a GPT-6-Astra Judge, using the same prompt and setup as MiMo for evaluation, to test consistency across models.

ComparisonEvaluatorsFirst-turn leakageSubsequent-turn leakageAnswer modificationOverall agreementCohen’s κ
Human–humanH1 vs. H280.0%80.0%80.0%80.0%0.600
Human consensus–MiMoConsensus vs. MiMo87.5%100.0%62.5%83.3%0.660
Model–model, full sampleMiMo vs. GPT98.57% (207/210)97.89% (1,813/1,852)95.29% (1,759/1,846)96.70% (3,779/3,908)0.930

Since this paper focuses its filtering solely on the disclosure metric, with answer modification serving only as an observational metric, human-model consistency on the two filtering metrics exceeds 90%, and inter-model consistency exceeds 97%. The results are relatively reliable and won't change the main experimental conclusions.

Thank you again for your constructive advice, especially valuable for helping us improve the rigor of our experimental implementation :) We hope the additional results and the responses above address your concerns.

Thank you again for raising the concern about experimental details and judge reliability.

We've built a dashboard for your convenience to view the updated experimental results, and added a case-by-case analysis of human–model labeling disagreements: https://relational-evidence-1c91bf.vercel.app/

We hope this additional evidence addresses your concerns about the labels, and we would be happy to provide any further clarification.

Thank you again for your thoughtful review. With less than a day remaining in the discussion period, we would appreciate your thoughts on our response and whether it addresses your concerns. If any concerns remain, we would welcome the opportunity to clarify them before the deadline :)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response from Authors

We thank the reviewers for their constructive feedback and for recognizing the value of the relatively underexplored aspect (by XuAe) & well-motivated question (by TLgN), the well-designed evaluation settings(by ah54) and effective definition separation of thought-privacy, self-consistency, and cross-order agreement(by all reviewers). We also thank XuAe for the positive assessment, who further expand the significance of this work to VLM scenarios, and TLgN for enhancing the solidity and reproducibility of the results.

Following the reviewers' suggestions, we have strengthened the evidence and revised the manuscript to address the concerns, the major updates are summarized below:

  1. Behavioral result validity and sample support(ah54, TLgN). For the noisy result, we identify the root cause, rerun the whole experiment and update Figure 3 using new results. Revised Figure 3 expands to 15 model settings, randomized query pair orientations that avoid constant-label shortcuts, provide ALL/Thought-private views, retained counts, and standard deviations (more details are specified in Appendices B.5 and C).

  2. VLM and visual memory(XuAe). Following the reviewer's suggestions, we add VLM scene reconstruction experiments on 40 visual scenes (Appendices F.2–F.3). These examine whether a generated record preserves accurate object relational information and whether image-conditioned text hidden state contributes beyond the token-level text information. This helps broaden the research questions to VLMs.

  3. Reproducibility(TLgN). We will open-source our code and data, including all the things the reviewer mentioned. Anonymous link for preview: https://anonymous.4open.science/r/Keep-In-Mind-LLM-77B5/

  4. Judge check(ah54). Appendix E.1 adds independent human annotations and a GPT cross-check, and we provide link for case study: https://relational-evidence-1c91bf.vercel.app/#validation. The consistency of human-model and inter-model consistency exceeds 90%, showing that results are relatively reliable and won't change the main experimental conclusions.

  5. Readout and replay interpretation(ah54, TLgN). Our reply to ah54 reports a direct-reveal positive control, and our reply to TLgN reports CoT-redaction results. These provide additional ablation and more fine-grained analyses of results.

We thank all reviewers for their feedback that materially improves our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.