acceptodds
Under review as a conference paper at ICLR 2027

Does the Workspace Do the Work? A Causal Protocol for Verifying Global-Workspace Computation in a Language Model

Abstract

Global Workspace Theory (GWT) is a leading theory of consciousness. It holds that information becomes conscious when it enters a limited-capacity workspace and is broadcast to specialized processes that use it. As AI is increasingly used to build and test such theories, an essential step is to determine whether a workspace built into a network is actually used or merely present. We introduce a causal protocol that answers this in stages. It asks whether the model depends on its workspace, whether receiving modules compute with the broadcast content, how that content acts, and whether it reaches several receivers at once. We apply it to a small language model whose modules can communicate only through a learned workspace. We confirm the results on four models trained under a protocol fixed in advance. Switching the broadcast off lowers accuracy by 41–52% on questions that need two modules' knowledge, while changing accuracy on single-module questions by at most 1.4 points. Dependence alone does not show use, so we then replace the broadcast with one taken from a different input. The receiver then combines the new content with its own memory, rather than copying the other input's answer, in 61–73% of cases, compared with 5–13% without intervention. Undoing only the resulting change in what the receiver retrieves restores the original output in 2,047 of 2,048 cases. This shows that the content acts through memory retrieval. A separately trained variant shows one broadcast serving three receivers. The protocol turns GWT's central claim into a testable property of trained models.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Global Workspace Theory, causal verification, mechanistic interpretability, activation patching, language models, model organisms

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dgVqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that building a Global Workspace Theory (GWT) component into a network does not show that the network uses it. It proposes a five-rung causal protocol: presence, dependence, content use, mechanism, and breadth. Its key tool is a transplant whose recipient, donor and hybrid answers are pairwise distinct, which separates ignoring, relaying and computing.

The protocol is applied to AFGW, a 6.9M-parameter model whose modules communicate only through a learned workspace, and confirmed on four models trained under a frozen protocol. Zeroing the broadcast cuts composition accuracy by 41–52 points, and a packet transplant yields the hybrid answer in 61–73% of cases versus 5–13% natively. Restoring the receiver's final-round queries recovers the native output in 2,047/2,048 cases. A post hoc query-only transplant and a separate three-receiver variant support sufficiency and breadth.

Critical strengths
  • Rigorous confirmation. Discovery is separated from a hash-committed, frozen protocol with fresh corpora and seeds; denominators are complete and every exception is reported.
  • Clean causal design. The three-value transplant separates ignoring, relaying and computing, and the 2×2 query design correctly distinguishes necessity from sufficiency.
  • Clear framing, honest scope. The verification gap and Proposition 1 explain why ablation is uninformative, and the claims are carefully limited.
  • Useful testbed. AFGW offers exact counterfactual ground truth for testing interpretability methods.
Critical weaknesses

W1. Architectural dependence of the mechanism. The tied-embedding head directly receives final lexical reads, favoring retrieval-based solutions. The direct-state route remains available, but the study does not establish whether the mechanism persists in a trained variant without this explicit lexical-read input.

W2. Limited GWT specificity. Competition and capacity are implemented, but their functional contribution remains untested, as Table 1 acknowledges. Routing weights π are not characterized, and no matched noncompetitive baseline establishes what the results show beyond general modular communication.

W3. Uneven confirmation strength. The query-only test reuses observed panels; the three-receiver variant uses development families and a different decoding schedule. These limitations are disclosed, but sufficiency and breadth have weaker confirmation than the primary restoration result.

W4. Control limitations. The same-facts/order control changes 13–32/128 outputs. Appendix H.4 explains order sensitivity analytically, but robustness to a permutation-invariant reader remains untested. Appendix F.1 also acknowledges that matched-norm noise controls magnitude rather than effects on retrieval scores; Figure 1’s “excludes query damage” overstates this control.

W5. Limited external validity and incremental novelty. The main evidence comes from a 6.9M-parameter synthetic setting with 64–75% composition accuracy. The discovery-stage eight-entity test is useful, but applicability to pretrained models remains untested. Novelty lies primarily in integrating established causal tools into a verification protocol.

Questions for authors
  1. Section 5.4 describes a single transplanted packet, whereas Appendix G.2 specifies replacements at writes 0, 1, and 2. Please reconcile these descriptions and explain the three-receiver variant’s 512/512 hybrid-output agreement.
  2. Does the receiver’s pre-read state h3h^3 retain information about alternative local answers that could support computation through route (b)?
  3. With receiver memory fixed, would queries targeting a third intermediate entity induce the corresponding hybrid answer? This would provide a structured semantic control beyond random noise.
4 comments
AuthorsOpenReview ↗

Response to Reviewer dgVq (1/2): W1, Q2, Q3, W3

Thank you for the careful review and for recognizing the frozen confirmation and the causal design. We answer in two comments and keep your labels. This one (1 of 2) covers W1, Q2, Q3 and W3, the points that the new experiments answer most directly. The next covers Q1, W2, W4 and W5.

W1. Retrieval persists without the direct lexical-read input. We trained four matched pairs of models with the original and the no-direct head, on the same richer training pool (24 more families) and budget (12,288 updates). All four no-direct models reach 97.83–100% held-out composition. Transplanting the source's writes 0–2 yields the hybrid answer in 503–509/512 cases per model, and restoring and installing the receiver's queries across reads 1–3 jointly reverse and reproduce 98.8–100% of changed outputs. The read effect now spans rounds 1–3 instead of round 3 alone, and the protocol locates this shift. On composition chains absent from training, restoration returns 546/547 changed outputs to native, against 2/547 for matched noise (§5.3, Appendix I.2).

Q2. Route (b) stays open in the test, and the output still follows the queries. With the transplanted broadcast in place, restoring the native queries returns the native output, and installing only the induced queries reproduces the transplant. Both hold in 1,551/1,555 changed cases, and the direct effect of the broadcast on the hybrid rate, with queries held native, is zero in every model (Appendices E.1 and H.3). On what h3h^3 contains: a ridge probe decodes the native answer from it at 57–71%. For alternative person-to-place assignments, cue plus h3h^3 gives 35.1–36.1%, against 34.7–35.0% for the cue alone (Appendix E.3). A probe cannot show that such information is absent, which is why the answer rests on the interventions.

Q3. Yes: queries induced by a third entity yield the corresponding hybrid answer. With the receiver's memory fixed, a second donor binds the object to a third person (Cara in the paper's example). Across the 272 cases selected in advance by symbolic conditions, installing only the queries that this donor's packet induces yields that person's location in the receiver's unchanged memory, the corresponding hybrid answer, in 200 cases, against 24 natively, and gives the same output as the full third-entity transplant in all 272 (Appendix E.5). The control also holds at high competence. Under the richer recipe of W1, the original-head models give the third person's hybrid answer in 572/586 cases, against 2/586 natively (§5.3, Figure 5c, Appendix I.1).

W3. The four-condition design replicates on unseen panels. We evaluated the four original checkpoints on two new panels, one per corpus, fixed before inference. They share no complete world or entity set with any earlier training, development or confirmation data. The transplant yields 1,365/2,048 hybrid answers, against 227 natively. Restoration returns 2,048/2,048 outputs to native, query-only installation reproduces the transplant in 2,044/2,048, and both hold in 1,530/1,534 changed cases (Appendix E.5). The three-receiver variant is also re-evaluated on four new families with a single packet (Q1, next reply).

AuthorsOpenReview ↗

Response to Reviewer dgVq (2/2): Q1, W2, W4, W5

This comment (2 of 2) continues our reply with Q1, W2, W4 and W5.

Q1. The submitted test replaced writes 0–2. The breadth test is now a single packet on new families.

What was done. The submitted experiment replaced the source's writes 0–2, as Appendix G.2 stated. Section 5.4 should not have called this a single packet.

What 512/512 meant. It counted agreement with the model's own outputs on the hybrid world, errors included, not correctness. The trajectory transplant replaces the source's packet at every write the receivers consume and leaves their memories untouched. Each receiver then decodes its whole answer from its private cache. The new panel reproduces the complete agreement, 512/512 episodes, while joint correctness is 458/512 and 416/512.

The new test. It uses the write-2 packet alone, donors whose maps differ from the recipient's, so that the donor's answer and the hybrid answer are distinct, and an unseen panel of 512 episodes from four new families. The single packet yields all three receivers' hybrid answers jointly in 448/512 and 412/512 episodes (native joint accuracy 458 and 416, accidental 20 and 12). Writes 0 or 1 alone stay at the accidental rate, and the donor's joint output is copied in 0/512 episodes. Per receiver, restoration recovers every changed output, query-only installation reproduces 96.8–100%, and patching one receiver leaves its siblings bitwise unchanged. The breadth claim is limited to this variant and its decoding schedule (§5.4, Table 2, Appendix I.3).

W2. Broadcast content is used without competition, and learned routing strengthens a single packet. Models trained with uniform routing still recombine through redirected retrieval. Restoration returns all 1,315 changed cases to native, and query-only installation reproduces 1,312. Learned routing adds control for a single packet: it yields 18–82 more hybrid answers per 512, while trajectory results stay similar.

For each of the four original models, the baseline is a model with routing fixed at 1/51/5 over the four modules and NULL, trained from the same starting checkpoint with the same data order and budget. Native accuracy differs from learned routing by at most 6/512. On the routing weights: at write 2 the learned router puts 0.956–0.995 of its weight on the source, and scaling that weight scales the transplant effect. Section 7 states that use does not depend on how access is allocated, and Table 6 records that it holds with learned and with uniform routing (Appendices E.6 and I.4).

W4. Each control limitation now has a direct test. Does the result depend on record order? In two models with a permutation-invariant reader, reordering records changes 0/512 outputs. Recombination remains: 392/512 and 426/512 hybrid answers, against 35 and 24 natively, and restoration and installation both hold in 424/424 and 442/442 changed cases (Appendix I.9). Is the query effect more than a perturbation? Matched-norm noise is now described as a magnitude control only, and the protocol figure no longer says it "excludes query damage". Two tests give specificity. Under the trajectory transplant, restoring the round-2 queries instead of the round-3 queries recovers 0–3 changed outputs, against 99.3–100% (Appendix E.3), so the effect is tied to the read site. The third-entity control (Q3) shows that the queries carry which entity to look up.

W5. A pretrained model, higher competence and larger worlds. Pretrained model. As a proof of concept, we ran the content and read tests in Qwen2.5-1.5B-Instruct. The message is a name the model generates and later looks up. Transplanting the name token's activation raises hybrid answers from 12/128 to 71/128, and restoring or installing attention rows, in a range of layers located from native attention, succeeds jointly in 70/72 changed cases, against 2/72 for matched random rows (§5.6, Appendix I.10). Competence. Under the richer recipe the original head answers 2,006/2,048 composition questions, and single-packet restoration at read 3 returns 534/534 changed cases on chains absent from training. The models of W1 compose at 97.83–100% and show the same mechanism (§5.1–5.3, Appendices I.1 and I.2). Larger worlds. Without retraining, the original four checkpoints carry the mechanism to eight-entity worlds: hybrid answers rise from 77 to 513 of 1,024, and both read tests hold in 798/804 changed cases (Appendix E.5).

On novelty, we agree that the tools are established. The contribution is what their combination distinguishes: relay and recombination systems on which accuracy and ablation give the same verdict, separated in 1,024/1,024 cases each (Appendix I.6), and content use on a route that zero-ablation calls dispensable (Appendix I.7).

We hope these results address your soundness concerns and support a stronger assessment.

I thank the authors for their detailed rebuttal and the additional experiments. The revisions address my concerns, and I appreciate the authors’ effort to improve the paper. I maintain my rating of 3 (weak accept).

AuthorsOpenReview ↗

Follow-up: one architecture learns opposite computations

Thank you for your reply, and for confirming that the new experiments address your concerns. The newest revision adds one finding that bears on the contribution, and it now reports its main results on the high-accuracy models.

New finding: one architecture learns opposite computations, and the protocol identifies which. Two AFGW models share architecture, starting checkpoint and training budget, and differ only in their training data. In one, the receiver's records are resampled at random during training, so only the source can compute the answer. Both answer at least 511/512 held-out composition questions, and both fail when the workspace is zeroed. A standard patch, whose donor answer is also the hybrid answer, moves both to the donor's answer in 512/512 cases. The three-value transplant gives the donor's answer in 511/512 cases in the first model and the hybrid answer in 512/512 in the second. The read tests hold in 512/512 changed outputs in the second model and in 0/511 in the first, so they do not pass where content does not act through retrieval.

A further workspace model, with a single memory reader, answers every held-out question of a two-hop narrative task while its broadcast goes nearly unused: zeroing everything its modules receive leaves 2,017/2,048 answers correct, and a donor's broadcast changes 0/2,048 outputs. The protocol therefore returns all three verdicts on trained workspace models: ignored, relayed and recombined. The division of work is learned, and accuracy, ablation and a standard patch do not reveal it (Figure 1, §5.5, Appendices I.4 and I.5).

The finding rests on high task competence (W5). The original-head models of W1 in our first reply are now the main models of the paper. They answer 2,006/2,048 held-out composition questions. One write-2 packet yields the hybrid answer in 1,990/2,048 cases, against 9 natively, and both read tests hold in 2,013/2,015 changed cases (§5.1–5.3, Appendix I.1). Without retraining, the same models answer 973/1,024 composition questions in eight-entity worlds, and both read tests hold in 1,005 of the 1,008 outputs that one packet changes (Appendix I.8). Breadth is reported with the same measure: both read tests hold in 2,894 of the 2,926 outputs that the single packet changes (§5.4). The frozen confirmation is unchanged. Section 4 states that its four models give the same verdicts, and Appendix E collects it with every further analysis of those models.

W2. Both routing rules support content use, and learned routing concentrates it. In a matched baseline at the same accuracy (2,001/2,048 against 2,006/2,048), all three packets yield the hybrid answer in 2,001/2,048 cases under uniform routing and under learned routing. A single packet yields it in 1,835/2,048 cases under uniform routing and in 1,990/2,048 under learned routing. Under this single-packet transplant, both read tests hold in 1,890/1,914 and 2,013/2,015 changed outputs. Use therefore does not depend on how access is allocated, and learned routing concentrates the content in one packet. We wrote that the commitments table marks the necessity of competition as not established. The table (now Table 6) and Section 7 state this finding directly (Appendices I.4 and E.6).

Some sections, appendices and tables are renumbered, and we have edited our earlier replies to you so that their statements and references match the revision. We hope the new finding supports a stronger assessment of the contribution.

Reviewer_dGHBOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2(+1)
Soundness
2
Read review
Paper summary
  • This paper investigates whether a language model equipped with a global workspace actually uses information exchanged through that workspace to perform computation. The work builds on Global Workspace Theory, which proposes that specialized processing modules communicate through a shared information channel. The authors highlight a critical distinction between having a communication mechanism, depending on it, and performing meaningful computations with the transmitted information. The paper proposes a causal verification framework for determining whether a neural network genuinely uses a shared workspace.
  • The authors test it on a test system, AFGW, which is a 6.9M-parameter model with four recurrent modules that communicate only through a learned, competitive 640-dimensional workspace.
  • The paper finds that the network depends on the workspace, where disabling communication decreases compositional-question accuracy while barely affecting local questions.
  • They also find that modules actually combine information from each other. When the authors replaced one module's message with information from a different example. The receiving module combined the new information with its own knowledge to produce the expected answer. This suggests the modules perform meaningful computation using shared information rather than simply copying answers.
  • I think the interesting finding is that the shared information changes what the receiving module retrieves from memory. The paper finds that a shared message influences which facts another module retrieves from its memory. When they restored the original memory queries, the model recovered its original answer and, similarly, changing only the memory queries reproduced the message-transplant result. This provides strong causal evidence that the workspace operates by redirecting memory retrieval.
Critical strengths
  • The experimental setup is pretty robust, with the protocols developed on separate discovery models and then evaluated on four newly trained models using a fixed protocol and new corpora.
  • The proposed research method is novel in how it is designed. The causal protocol could inform studies of communication between modular neural networks, retrieval systems, and specialized agents.
  • The experiments provide strong evidence for the proposed mechanism. In particular, restoring the original memory queries recovers the original outputs in 2,047 out of 2,048 cases, which supports the claim that shared information influences memory retrieval.
  • The paper includes several control experiments, such as random perturbations, self-replacement, and interventions at unused communication steps. These controls help rule out alternative explanations for the observed behavior.
Critical weaknesses
  • I guess since the experiment setup is so well engineered, I have a small doubt that the architecture is deliberately engineered to make information flow identifiable. The workspace is the only communication route between the modules. Hence, the dependence of compositional tasks on the workspace follows directly from the architecture. The experimental setting is considerably simpler than a conventional Transformer, where information can travel through multiple overlapping pathways.
  • Similar to the architecture design, the tasks are highly controlled and synthetic. The experimental worlds contain small collections of objects, people, and attributes. Such environments support exact counterfactual construction, but they limit evidence about whether the approach extends to natural language reasoning or complex learned knowledge.
  • Since the experiments use a specifically designed small model (6.9 million parameters) trained on synthetic questions, I am not sure if it would hold the same for a large pretrained model.
Questions for authors
  1. Does the proposed verification protocol require an exclusive communication pathway and identifiable memory queries, or can its causal conclusions remain valid without these architectural constraints?
3 comments
AuthorsOpenReview ↗

Response to Reviewer dGHB

Thank you for your strong support and for the precise question. The new experiments give a direct answer: a route can contribute content to the receiver's computation even when another route sustains performance after its removal. We answer your question first, then the three concerns.

1. Which conclusions need an exclusive pathway and identifiable queries? Each test has its own requirement (Table 5, Section 7).

  • Dependence. Ablation measures the effect of deleting one route. Other routes can compensate, so a small deficit does not imply little use. The memory-only bound of Proposition 1 holds when the workspace is the receiver's only cross-module input.
  • Content use. The transplant needs an identifiable carrier and counterfactual inputs with distinct predicted answers. It does not need an exclusive pathway.
  • Mechanism. The read tests need a place where the receiver's read can be intervened on. This is the memory query in AFGW and an attention row in our Transformer experiment. Restoration puts back the receiver's original read and tests whether the changed read is necessary for the effect. Installation puts in only the changed read and tests whether it is sufficient.

2. Overlapping pathways, tested directly. We trained two dual-route models with a learned bypass between the memory readers, with the workspace zeroed on half of the training rows so that either route can sustain performance. Zeroing the workspace costs only 22 and 6 of 512 answers, against 244 and 216 when both routes are zeroed, so zero-ablation alone would call the workspace nearly unnecessary. Yet transplanting its content creates 76 and 134 new hybrid answers, against 3 and 2 for noise of the same size. Both read tests hold in 132/136 and 173/173 of the outputs the transplant changes: restoration undoes the change and installation reproduces it. Restoring the same read also reverses the bypass's transplant effects (265/266 and 154/154), so both routes are consumed through one read (§5.6, Figure 6a, Appendix I.7).

3. What is engineered and what is learned. Two variants test whether use of the workspace follows from the architecture. In the first, the receiver reads both memories, so it has all the information needed to answer alone and the workspace is optional. The trained models still lose 214–246 of 512 answers when the workspace is zeroed (Appendix E.7). The second removes the output head's direct lexical-read term, which could otherwise favor retrieval. Those models reach 97.83–100% held-out composition, and both read tests, applied across reads 1–3, hold for 98.8–100% of the outputs the transplant changes (§5.3, Appendix I.2). The architecture fixes which information each module can access. What the channel carries and how the receiver uses it are learned, and these are what the content and read tests measure. In the main models, the direct route from the broadcast into the receiver's state is open in every round, yet the output follows the receiver's queries.

4. Synthetic tasks and scale. In a pretrained model, as a proof of concept, we ran the tests in Qwen2.5-1.5B-Instruct. The message is a name the model generates, which a later step looks up in a name-to-code table. The packet transplant becomes a transplant of the name token's activation, and the query intervention becomes restoring or installing attention rows, with value vectors computed normally. Hybrid answers rise from 12/128 to 71/128, and both read tests hold in 70 of the 72 outputs the transplant changes, against 2/72 for random rows of matched size (§5.6, Appendix I.10). Ablating the name removes only part of the model's accuracy, so the tests run here with overlapping pathways.

In the controlled worlds, where every counterfactual target is exact, the results replicate on unseen evaluation sets (both read tests hold in 1,530 of the 1,534 changed outputs), carry over without retraining to worlds twice as large (798/804), and hold in point-to-point and slot-workspace models (Appendices E.5 and I.9).

5. Presentation. The protocol is now introduced through one running example, the football question, and the figures are redrawn around it. Table 1 collects the key terms, and Table 2 aligns each intervention with what it does not intervene on and the prediction it tests.

We hope these results strengthen your confidence in the paper.

AuthorsOpenReview ↗

Follow-up: what is engineered and what is learned

Thank you again for your strong support. The newest revision adds results on your first concern, that dependence on the workspace follows from an architecture engineered to make information flow identifiable. It also reports its main results on models with high task accuracy.

What the engineered boundaries are for. We trained a workspace model with the same kinds of component, four recurrent modules, four rounds and a competitive broadcast, but with a single memory reader and an output head that also reads the workspace. Two such models answer every held-out question of a two-hop narrative task. Zeroing everything their modules receive from the workspace leaves 2,017/2,048 answers correct, and a donor's broadcast changes 0/2,048 outputs. Without AFGW's two boundaries (split memories, no path from the workspace to the output), a trained broadcast goes nearly unused (§5.5, Appendix I.5).

What training decides. Across AFGW and its matched variants, the experiments establish four learned properties.

  • How strongly the model depends on the workspace. With the workspace off, the main models answer 21–27% of composition questions, although the receiver's own memory supports 46–53% (§5.1).
  • Whether an optional workspace is used. When the receiver reads both memories and could answer alone, four models answer 2,002/2,048 composition questions, and zeroing the workspace leaves 458/2,048 correct (Appendix I.4).
  • Who computes. Two models with the same architecture, starting checkpoint and budget differ only in their training data. One relays an answer computed by the source, and the other recombines the message with the receiver's memory. Both answer at least 511/512 held-out questions, and both depend on the workspace. Under the three-value transplant the first gives the donor's answer in 511/512 cases and the second the hybrid answer in 512/512 (§5.5, Appendix I.4).
  • How the content acts. The direct route from the broadcast into the receiver's state is open in every round, yet the output follows the receiver's queries in 2,013/2,015 changed cases (§5.3).

Stronger models and larger worlds. The main text now reports four models that answer 2,006/2,048 held-out composition questions. One transplanted message yields the hybrid answer in 1,990/2,048 cases, against 9 without intervention (§5.1–5.3). Without retraining, these models answer 973/1,024 composition questions in eight-entity worlds, and both read tests hold in 1,005/1,008 of the outputs the transplant changes (Appendix I.8). The four models of the submitted version give the same verdicts, and Appendix E collects them.

Presentation. Section 5 now runs the four tests in order, each result is reported under one condition, and Table 3 in the main text lists the models.

Some sections, appendices and tables are renumbered in this revision, and we have edited our earlier reply to you so that its references match it.

Thank you for the detailed response and the additional experiments. The dual-route and boundary-free experiments address my main concern, and the Qwen result is a useful proof of concept. Please note in the limitations that workspace use depends on the engineered boundaries and that larger models require first locating a carrier and read site. I am keeping my score.

Reviewer_CLxBOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2(+1)
Soundness
3
Read review
Paper summary

This paper studies global workspace theory in language models. The authors claim that there exists a "verification gap" in which prior work has used language models to construct models of cognitive theories but lack processes to verify whether the constructed model is using the model in the intended way. The authors present three ways to perform this verification. They then train their own model organism (AFGW) to demonstrate their framework.

Critical strengths

None

Critical weaknesses

It is not clear to me that the contribution of the paper is novel. It seems like the paper is just performing progressively targeted and fine-grained activation patching. The tradeoffs between these types of patching has been extensive discussed in works like ZN23.

Along these lines, it is unclear to me that there exists a "verification gap." I think the motivation of the paper could be made a lot more strong if the authors provided examples of existing models of global workspace theory where the language model is "hacking" the broadcasting / communication channels. To me, it is hard to imagine that a model could do this, given that the modules and their interactions are so rigid.

My other concern is the generality of the findings here. The authors train their own model and then verify using their framework that the model passes all three tests. But, the test themselves are hyper-specific to the model they create. It is unclear to me how to generalize to test say whether a general language model is implementing a global workspace [GSPP+26].

I also found the paper very difficult to follow. There were many instances where notation was being referenced without being introduced a priori. Further, I found it hard to understand what was truly different between the levels of interventions.

Questions for authors
  • I'm confused by the claim in Proposition 1 (even more so when I read the proof in Appendix F). It would help my understanding if the authors could state this precisely using the formalisms introduced in the paper (what is OFF?).
3 comments
AuthorsOpenReview ↗

Response to Reviewer CLxB (1/2): novelty and the verification gap

Thank you for finding the work technically sound and for asking us to make the contribution concrete. We answer in two comments. This one (1 of 2) covers novelty and the verification gap. The next covers generality, Proposition 1 and presentation.

1. What is new beyond activation patching. Two systems can agree on accuracy, on dependence on communication, and on whether a patched message changes the output, while the source computes the answer in one and the receiver computes it in the other. Our contribution is a calibrated protocol that tells these cases apart and then tests how the message acts in the receiver. The source may send a finished answer that the receiver forwards (relay), or the receiver may combine the message with its own knowledge (recombination). The interventions themselves are established, and we build on activation patching and interchange interventions. The protocol combines three elements.

  • A three-value transplant. The swapped-in message is chosen so that ignoring it, relaying it and recombining it predict three different answers. In the common two-value design, relay and recombination predict the same string.
  • Calibration. The test is run on reference systems whose computation is known (point 2).
  • Read interventions. Restoring and installing the receiver's read queries test whether the message acts by redirecting retrieval.

Example: in the recipient input, Jeff holds the football and is in the room, and Bill is in the hospital. In the donor input, Bill holds the football and is in the kitchen. We transplant the donor's message about the holder and keep the recipient's location memory. Ignoring the message predicts "room", relaying the donor's answer predicts "kitchen", and recombination predicts "hospital", which we call the hybrid answer.

Zhang & Nanda ask how to patch reliably, and we follow their advice. Composition accuracy drops under zero, mean and resample ablation, and exact match and log-probability agree on all 32 transplant verdicts (Appendix E.3). Our question is a different one, who computes with a message. §6 cites the precedents for distinct-target designs and states what we add.

2. Is the verification gap real? Yes. A fixed interface does not fix what is sent through it, and published modular systems show this. In emergent communication, the answering agent conveys the whole instance in a code of its own and mostly ignores the question, with near-perfect reward (Kottur et al., 2017). In rationale models, the predictor sees only the selected text, yet the selection can become a code for the label (Yu et al., 2019). In concept bottlenecks, soft concept scores carry task information beyond the concepts (Havasi et al., 2022). These examples come from other modular systems. Trained workspaces have so far been evaluated by performance, ablation and transfer (§6, Appendix B.1). Such checks establish task success and dependence on communication. They do not show whether the source relays a finished answer or the receiver recombines.

Rigid modules do not rule out relay. Typed memories fix what each module reads directly. Over several rounds, the source can still receive the location facts through the broadcast and send the finished answer itself. The architecture therefore leaves open who composes.

Two reference designs make this concrete. Their architectures fix the computation, one to relay and one to recombine, which gives known cases on which to calibrate the test (two seeds each). All four answer 1,024/1,024 held-out questions correctly, all depend on their channel, and in all the transplant changes the output. These checks give the same verdict for opposite computations. The three-value transplant separates them: both relay models give the donor's answer and both recombination models give the hybrid answer, in 1,024/1,024 cases each (Figure 1, §3.2, Appendix I.6).

The three-value test finds recombination in the four models of the submitted version and in twelve variants of them, including variants in which relay or a purely local solution was available. A direct check that alters one module's memory agrees in every one (Appendix E.7).

Necessity and content use are also different properties, and ablation measures only the first. The two dual-route models have a learned second route between the two memory readers and were trained with the workspace zeroed on half of the examples, so either route can sustain performance. Zeroing the workspace costs only 22 and 6 of 512 answers. Transplanting its content still moves 76 and 134 outputs to the hybrid answer, against 3 and 2 for noise of the same size (§5.6, Appendix I.7). The workspace is largely replaceable, and the receiver still computes with what it carries.

AuthorsOpenReview ↗

Response to Reviewer CLxB (2/2): generality, Proposition 1 and presentation

This comment (2 of 2) continues our reply with generality, Proposition 1 and presentation.

3. Generality. What transfers is the experimental contrast, not AFGW's modules. Keep the receiver's own information fixed, change the communicated content so that the candidate computations predict different answers, and intervene on what the receiver reads.

At another interface. Qwen2.5-1.5B-Instruct shows these steps in a pretrained model, as a proof of concept. The message is a name the model generates, which a later turn looks up in a table of names and codes. We transplant the activation of that name token from another input. The answer that combines the transplanted name with the model's own table (the hybrid answer) rises from 12/128 to 71/128. The read intervention acts on the attention weights of the answer position, in a range of layers located from native attention. The transplant changes the output in 72 cases. In 70 of them, putting back the original attention weights undoes the change, and installing only the new ones reproduces it. Random attention weights of matched size do both in 2 (§5.6, Appendix I.10).

Across design choices. The same result holds in trained variants that differ from AFGW in routes, output head, routing rule, memory reader or communication design, including a slot-based workspace in the style of Goyal et al. In every variant the transplant raises hybrid answers. In at least 95% of the outputs it changes, restoring the receiver's original read undoes the change and installing only the new read reproduces it (Figure 6, §5.6). The result therefore does not require an exclusive channel, a direct lexical read in the output head, learned routing, an order-sensitive reader or AFGW's broadcast design.

From discovery to verification. Gurnee et al. identify workspace-like representations in pretrained models and study the roles they play downstream. Our protocol adds two things to such work: comparisons calibrated to tell relay from recombination, and paired interventions on the reader. The Qwen study shows the steps: identify what the candidate carries, locate the positions that read it, build inputs on which the candidate computations predict different answers, then run the transplant and the read interventions.

4. Proposition 1 and OFF. OFF sets to zero every broadcast received by every module, in every round and at every generated token (Definition 1, §3.1). With deterministic decoding, independently encoded records and no other cross-module input, the receiver's complete answer under OFF is Fθ(Q,Bo)F_\theta(Q,\mathcal{B}_o), a function of the question QQ and its own memory Bo\mathcal{B}_o only. Two inputs that agree on both receive the same OFF answer, so if their correct answers differ, at most one is right. This is the memory-only bound (Appendix F.1). The observed drop is 69–78 points for composition and at most 2.3 for local questions in the main models, and 41–52 and at most 1.4 in the frozen confirmation. The proposition states what the receiver can know without communication. The later tests show what the communicated information does.

5. Presentation. The protocol is now introduced through the football example. Table 1 collects its key terms, and Table 2 aligns each intervention with what it does not intervene on and the prediction it tests. OFF asks whether communication is needed. The transplant asks what its content contributes. Restoring and installing the receiver's queries ask how that content acts.

We hope these results make the contribution concrete, and we would welcome your reassessment.

AuthorsOpenReview ↗

Follow-up: a standard patch cannot tell relay from recombination

Good morning and thank you again for your constructive comments! The newest revision adds a direct test of your first concern, that the method is progressively targeted activation patching. For your second concern, it adds trained workspace models that do not use the channel as intended.

A trained AFGW model relays, and a standard patch cannot show it. Section 5.5 has two AFGW models with the same architecture, starting checkpoint and training budget. In both, the source can read both memories. In the first, the receiver's records are resampled at random during training, so only the source can compute the answer. In the second they are intact. Nothing in the architecture or the messages is set to relay, and at test time both models see intact memories. We patched the same site in both models with two kinds of donor, on the same 512 held-out inputs.

  • Standard donor. As in ordinary activation patching, the donor's answer is also the hybrid answer. The output becomes the donor's answer in 512/512 cases in the first model and in 512/512 in the second.
  • Three-value donor. The donor's answer and the hybrid answer differ. The first model gives the donor's answer in 512/512 cases, and the second gives the hybrid answer in 512/512.

The patch, its site and the two models are identical in both rows. The standard design returns the same result for two opposite computations. Accuracy (512 and 510 of 512) and ablation (90 and 119 of 512 correct with the workspace zeroed) agree as well. Replacing one module's records at inference confirms both verdicts: the first model's answer follows the source's copy (1,168/1,171 cases), and the second's follows the receiver's memory (1,171/1,171). What lets a patch separate the models is the choice of donor, which makes relaying and recombining predict different outputs. This choice, its calibration on systems whose computation is known, and the read tests are what the protocol adds to patching (§5.5, Appendix I.4).

A fixed architecture does not fix who computes. You found it hard to imagine a model misusing the channel when the modules and their interactions are so rigid. We repeated the training of the first model with the records resampled on only 25% of the examples and intact on the rest, on two initializations of each of our two corpora. All four new models relay. The three-value transplant gives the donor's answer in 428–512 of 512 cases (at least 508 in two of the models) and the hybrid answer in at most 44. When records are replaced at inference, the output follows the source's copy in 4,441/4,650 draws and the receiver's memory in 172. A standard patch again moves them to the donor's answer (473–512 of 512), as it does the recombining control. All of these models share one architecture. Records that were unreliable on one training example in four were enough for the source to take over most or all of the computation (Appendix I.4).

A built-in broadcast that goes nearly unused. We trained a workspace model with four recurrent modules, four rounds and a competitive broadcast, in which a single module reads all records and the output head also reads the workspace. Two such models answer every held-out question of a two-hop narrative task, half of them compositions. Zeroing everything the modules receive from the workspace leaves 2,017/2,048 answers correct, and replacing the broadcast in all four rounds with a donor's changes 0/2,048 outputs. The single reader chains its own reads, so nothing needs to cross between modules. Accuracy gives no sign of this (§5.5, Appendix I.5).

Three verdicts on trained workspace models. The same test now returns each of its three answers on a trained model. The broadcast is ignored (the model above), relayed (the five AFGW models trained to relay), or recombined (the main models, 97.2% hybrid answers at 98% task accuracy). Figure 1 shows the three beside the two reference systems.

Easier to follow. You found the paper hard to follow, and we have simplified it further. Section 5 runs the four tests in order (§5.1–5.4), then shows what training can produce (§5.5) and how far the tests travel (§5.6). Definition 1 and Proposition 1 state OFF and the memory-only bound in §3.1, and Table 3 groups the systems by the question each answers.

Some sections, appendices and tables are renumbered in this revision, and we have edited our earlier replies to you so that their references match it. We hope these results answer both concerns, and we would welcome your reassessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: summary of new results and revisions

We thank the reviewers for their careful and constructive reviews. The paper asks who computes when the modules of a network communicate. The revision adds experiments on the main points raised, and four results make the answer concrete.

The test in one example. One module, the source, knows who holds each object. Another, the receiver, knows where each person is. Asked "Where is the football?", the receiver may combine the source's message with its own memory (recombination), or it may forward an answer that the source has already computed (relay). Accuracy and ablation cannot tell these apart. We therefore replace the source's message with its message from another input, the donor. The donor is chosen so that ignoring the message, relaying it and recombining it predict three different answers: the original answer, the donor's answer, and a third that we call the hybrid answer. This is the three-value transplant. Two read tests then act on the receiver's memory queries. Restoring the original queries should undo the transplant's effect, and installing only the new queries should reproduce it.

1. Equally successful systems can compute in opposite ways, and the protocol tells them apart. Reference systems built to relay and systems built to recombine are all fully accurate, and all depend on their channel. Under the three-value transplant, every relay system gives the donor's answer and every recombination system gives the hybrid answer, in 1,024/1,024 cases each (Figure 1, §3.2, Appendix I.6).

2. In a learned workspace, shared content redirects what the receiver retrieves. On evaluation sets not used in any earlier run, restoring the receiver's original queries returns 2,048/2,048 outputs to their original values, and installing only the queries that the transplant induces reproduces the transplant in 2,044/2,048. The queries carry which person to look up. When a second donor's message names a third person, installing only the queries it induces yields that person's location in the receiver's memory in 200/272 cases, against 24/272 without intervention (Appendix E.5).

3. A replaceable route can still contribute to computation. In two models trained with a second route between the modules, zeroing the workspace costs only 22 and 6 of 512 answers, yet transplanting its content creates 76 and 134 new hybrid answers, against 3 and 2 for noise of the same size (§5.6, Appendix I.7). Ablation measures necessity. The transplant and the read tests measure use.

4. The finding holds at high task competence and with a different output head. With a larger training pool and longer training, four AFGW models answer 2,006/2,048 composition questions (98%). Where relay and recombination predict different outputs, the transplant gives the hybrid answer in 1,969/1,994 cases and the donor's answer in 12. On questions whose chain of object, holder and location never occurs in training, restoring the receiver's queries undoes 534/534 of the outputs the transplant changes. In matched models the output head sees the receiver's final state but no longer sees what it retrieved (the direct lexical-read term is removed). These models compose at 97.83–100% on held-out worlds, and restoring and installing the queries across reads 1–3 together account for 98.8–100% of the changed outputs (§5.1–5.3, Appendices I.1 and I.2).

The original models show the same pattern, without retraining, in worlds twice as large (Appendix E.5). It also holds under uniform routing (E.6), in point-to-point, slot-based and order-invariant designs (I.9), with one message serving three receivers (I.3), and, as a proof of concept, at a name that Qwen2.5-1.5B-Instruct generates and later looks up (I.10).

What changed in the paper.

  • The introduction, Figure 1 and Section 3 present the method through the football example before any formalism. Table 1 collects the key terms, Table 2 states what each test changes and predicts, and Table 5 states what each test requires.
  • OFF, the intervention that zeroes every broadcast, and Proposition 1 are restated precisely (§3.1).
  • The breadth test now uses a single message and separates relay from recombination (§5.4).
  • Figures 1 and 3–6 are redrawn. Changed passages are in blue. Appendix A maps each question to its evidence, and Table 4 lists the new experiments by reviewer point.

The replies below give the details for each reviewer.

AuthorsOpenReview ↗

New results and a clearer organization

We have uploaded the newest revision. It adds experiments completed since our first response and reports its main results on models with high task accuracy. Three things are new.

1. Trained workspace models compute in three different ways, and the protocol identifies each.

  • Relayed. Two AFGW models share architecture, starting checkpoint and training budget, and differ only in their training data. In one, the receiver's records are resampled at random during training, so only the source can compute the answer. Both answer at least 511/512 held-out composition questions, and both fail when the workspace is zeroed. A standard patch moves both to the donor's answer in 512/512 cases. The three-value transplant separates them: the first gives the donor's answer in 511/512 cases and the second the hybrid answer in 512/512. Replacing either module's records at inference confirms both verdicts (§5.5, Appendix I.4).
  • Ignored. We trained a workspace model with four recurrent modules, four rounds and a competitive broadcast, in which a single module reads all records. Two such models answer every held-out question of a two-hop narrative task. Zeroing everything their modules receive from the workspace leaves 2,017/2,048 answers correct, and a donor's broadcast changes 0/2,048 outputs (§5.5, Appendix I.5).
  • Recombined. The main AFGW models give the hybrid answer in 97.2% of cases (§5.2).

Having a workspace therefore does not settle what a trained model computes with it, and the protocol does (Figure 1).

2. The mechanism holds at 98% task accuracy and in larger worlds. The main text now reports, throughout, the four high-accuracy models introduced in our first response (2,006/2,048 held-out composition questions). One transplanted message yields the hybrid answer in 1,990/2,048 cases, against 9 without intervention. Restoring the receiver's queries returns 2,047/2,048 outputs to their original values, and installing only the new queries reproduces the transplant in 2,047/2,048 (§5.1–5.3). Without retraining, these models answer 973/1,024 composition questions in eight-entity worlds, and both read tests hold in 1,005/1,008 of the outputs the transplant changes (Appendix I.8). Matched variants with uniform routing, or with one module reading both memories, recombine at the same accuracy (Appendix I.4).

3. The paper is organized around this argument. Section 5 runs the four tests in order (§5.1–5.4), then shows what training can produce (§5.5) and how far the tests travel (§5.6). Table 3 lists the systems by the question each answers. The four models of the submitted version give the same verdicts (Section 4), and Appendix E collects them, unchanged, with every analysis run on them. Some sections, appendices and tables are renumbered, and we have edited our earlier comments so that every reference in them matches this revision.

Short follow-ups to each reviewer give the details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.