acceptodds
Under review as a conference paper at ICLR 2027

TEXT AS A COMPOSITIONAL INTERFACE: TOWARD DECOMPOSABLE LEARNING WITH RECON- STRUCTED POLICIES

Abstract

Complex systems should not require every new combination of learned skills to be trained end to end. We study whether a language model can instead externalize a learned component as a local textual policy and use that text as a runtime interface for composition. Models are fine-tuned on short chains of opaque triggers, each denoting a two-branch conditional policy; the definitions are absent from the test prompt, and the model reconstructs the complete policy immediately before each use. We then ask which parts of the reconstructed text actually control the next computation. Exchanging only the two operations inside a written policy, with the upstream trace and state fixed, redirects the next operation on the majority branches of both LLaMA-3.2-3B-Instruct and Qwen2.5-3B-Instruct (82–85% pooled), but far less on rare branches. On a separate customer-service workflow, editing a written threshold on the same record produces a +64.2 percentage-point decision-directed effect beyond an equally sized decision-preserving edit (95% CI 59.4–68.9), and the redirected decision propagates to the final verdict. The interface is not uniformly semantic, however: writing an unfamiliar modular-arithmetic residue into a policy yields no value-specific discrimination (−0.35 points), which bounds what operation following can certify and leaves open whether the bottleneck is reading the predicate or computing it. Finally, a token- and position-matched control shows that informative policy content remains useful at a higher numeric range (+23.3 points, 6/6 seeds), with a six-point attenuation at the held-out depth. Together, the results distinguish policy reconstruction, interpretation, action selection, and continued invocation as requirements on a textual interface, and motivate reconstructed policies as a route toward decomposable learning rather than a demonstration of it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
compositional generalization, language models, modular learning, rule following, scratchpad supervision, textual policy interfaces, behavioral interventions, reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZHuQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tests whether models can reconstruct learned policies as text and use that text to guide later actions. Its main strength is the careful separation of reconstruction, interpretation, execution, and invocation through controlled interventions. The results show strong operation following, but weak interpretation of new predicate values and limited reliability across branches. Overall, the paper provides a useful evaluation framework, but only preliminary evidence for decomposable learning.

Critical strengths
  1. The paper clearly separates reconstruction, interpretation, execution, and invocation. This provides a useful framework for evaluating textual policy interfaces.

  2. The controlled interventions identify both successful operation following and failed predicate interpretation. This supports careful and well-bounded conclusions.

Critical weaknesses
  1. Training includes all single triggers and ordered pairs of distinct triggers. Testing uses six permutations of three distinct triggers. This provides limited evidence beyond the observed training structures. Testing a larger trigger set or greater held-out depth would strengthen the evaluation.

  2. The novel predicate-value intervention shows −0.35 percentage-point matching-versus-off-target discrimination. This weakens the claim that the model interprets newly written predicate values. A direct-rule capability probe could identify the cause.

  3. Pooled operation-swap following is 82.0% for LLaMA and 85.2% for Qwen. However, two rare LLaMA branches achieve only 4.5% and 2.0%. This shows strong variation across branches. The branch-specific results should be emphasized.

  4. Preserving edits cause 16.1% decision churn. Thus, edits that preserve the prescribed decision for a record can still change the model’s decision. Further controls could examine this instability.

  5. Only 71.5% of saved free runs reach the canonical editable prefix. All qualifying prefixes exactly match the gold prefix. Thus, editing after noncanonical reconstruction remains untested.

  6. With informative policies, held-out depth-3 accuracy is 38.8% for starts 51–70 and 31.0% for starts 351–370. The paper should foreground these absolute results alongside the D–N advantage.

Questions for authors

Check my comments in Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZHuQ

Plain-language response (part 1 of 2: setup and points 1–3)

Thank you for telling us the response was hard to read. Our earlier reply had to fit six answers into one 5,000-character comment, and it became too compressed. Here we restate it in plain language, with more detail on what each test did. There are no new results; every number is from our earlier reply or the paper.

The setup. The paper has two tasks.

Arithmetic task. We fine-tune a model on short programs. Each step is named by a code letter (X, Y or Z), and each letter stands for a two-branch rule, for example X = "if the number is odd, add 1; otherwise add 2". The test prompt gives only the letters, never the rules. Before each step the model writes the rule out from memory, then applies it:

t2=X rule={odd:+1 | even:+2} pre=56 branch=even op=+2 post=58

We call this model D. As a control, model N is trained to write a meaningless text of the same length in the same place (rule={q:v | w:zzz}). Comparing D with N tells us whether the content of the rule helps, rather than just having extra text.

Workflow task. A separate model handles customer returns through four checks (e.g. "return window: days<=15"), and writes each check before applying it.

Our main tool is editing the written rule and seeing whether the model's next step follows the edit. If it does, the text is really being used.

1. "The test only recombines pairs seen in training." You are right: training included every ordered pair of letters, so a three-step test mostly chains familiar pairs. To test an unseen pair, we retrained with one pair (Y followed by X) removed.

  • The model then almost always fails on Y→X: D is correct 4.33% of the time, N 3.67%. In 248 of 300 outputs, D writes the rule of letter Z where X's rule should be. It recalls the wrong rule.
  • On a separate set of 50 test cases per model, we replaced that wrongly written rule with the correct one from a list of all rules, without telling the model which branch to take. Accuracy rose from 3.7% to 96.7%. So the model can use a correct rule; recalling it is what fails.
  • This is repair with help from us (the rule list), not composition by the model alone.
  • The same pattern held when we removed each of the other five pairs in turn (each ≤0.7% correct without help), and in a larger 8B model (23 of 300 correct; 2.3% → 98.3% with the repair).

For more letters and longer programs, we trained an 8B model with four letters on one- and two-step programs only. It solves 93.7% of three-step and 94.9% of four-step programs. However, every neighbouring pair was trained there, and it had more training data (1,600 vs 900 examples), so this shows longer chains of familiar pairs, not unseen pairs.

2. "The model does not understand new condition values." We agree; this is a limit of the paper. In the test, at a step where the number is 58, we rewrote Z's condition to "remainder after dividing by 5 equals 3" (true for 58) or "equals 2" (false for 58). If the model read the condition, it would take the first branch much more often in the "equals 3" case. It took it in 1 of 864 cases versus 4 of 864: no difference (−0.35 points).

As you suggested, we then tried a direct test of whether the model can compute such conditions at all. No model passed that test's own quality check (0 of 10, 0 of 6, 0 of 6 models), so we still cannot say whether the model fails to read the condition or fails to compute it. One finding helps: a condition the model already knows (from another letter) is followed when the whole rule is written out (balanced accuracy .963 and .977, where .5 means guessing), but not when it is mixed into the host letter's rule (.495 and .531).

3. "Rare branches." Swapping the two operations in a written rule (e.g. "+1" and "+2") changes the model's action 82.0% (LLaMA) and 85.2% (Qwen) of the time overall, but only 4.5% and 2.0% on LLaMA's two rare branches, which have 33 and 17 test cases per model. We will put these per-branch numbers in the main table. We also retrained with the rare branches shown more often. Swap following changed by −3 points and unedited accuracy on rare branches by +13 points; neither change is clearly different from zero, and accuracy on common branches fell by 11 points. So we no longer say rarity is the cause.

(continued in part 2)

Thanks for the rebuttal. It is very hard to read and understand the response which is consistent with my earlier observation with the paper (hence a score of 1 for presentation).

Can you please simplify your response (and paper) a bit? Please add details. I am not against the usage of LLM for writing but please make sure that a human can read and comprehend it.

Plain-language response (part 2 of 2: points 4–6 and summary)

4. "Edits that should not change the decision still change it 16.1% of the time." In the workflow test we take one customer record, for example days=18, and edit the written threshold in two ways that move it the same distance from 18: days<=15 (the correct decision should change) or days<=21 (it should not). Comparing the two on the same record removes the effect of "any edit at all".

  • The edits that should change the decision do so 64.2 points more often than those that should not.
  • But the "should not change" edits still change 16.1% of decisions. Newly trained models show the same rate (16.0%), so this instability is real, and we will state it plainly.
  • You asked for further controls. In the original test, the decision-changing edit usually moved the threshold further from its trained value (272 of 288 records). When we compare only edits matched on direction and size, the gap is still 49.9 points. One difference remains uncontrolled: how far the threshold moves from its trained value.

5. "Only 71.5% of the model's own runs could be edited, and all of those were already correct." Without our help, the model wrote a well-formed, editable workflow in 71.5% of runs, and in every one of those the text was exactly correct. So the original test never edited a run that had already gone wrong. We added that test with newly trained models (480 records each).

  • When an earlier written check is wrong but still editable (46, 53 and 33 such runs for the three models), we correct it at the known spot. Correct final answers rise by 90.2 points on those runs.
  • Across all records, including runs we could not edit, the gain is 8.3 points.
  • We tell the model where the error is and what the right check is, so finding and fixing errors on its own remains untested.

6. "Low absolute accuracy." We will report the absolute numbers first, not only the difference from the control. On new three-step programs, D is correct 38.8% of the time with small starting numbers and 31.0% with larger ones; N is correct 9.5% and 7.7%.

  • The help from the rule text depends on output format. When the model writes out its working, D beats N by 54.17 and 49.31 points; when it answers directly, the difference is only +4.31 and −2.78.
  • A larger 8B model, trained on all pairs, reaches 78.5% on three-step programs. But swapping the order of the two branches inside a rule, which means the same thing and was never seen in training, costs it 30.4 points. This is another limit we will state.

Summary: what we claim and what we do not.

  • The written rule does steer the model's next step for operations it already knows, in two model families, and in the workflow task an edited threshold redirects the decision.
  • When the model recalls the wrong rule, giving it the right rule restores most of the accuracy.
  • We do not claim that the model understands new condition values, composes pairs it never saw together without help, or is robust to rewording.

So we see the paper as a careful way to find where a written-rule interface works and where it breaks, not as proof that it works. In the final version, we will rewrite the paper in the same plain style as this reply, starting with the introduction and Section 5. If any part is still unclear, please tell us which one and we will explain it further.

Reviewer_EZJGOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether language models can use locally reconstructed text as a compositional interface. In a controlled setting where opaque triggers denote two-branch policies, models are fine-tuned on depth 1-2 chains and tested on depth 3 without rule definitions. The authors decompose the interface into reconstruction, interpretation, execution, and invocation. They find that while written operations can redirect next-step computation (majority branches), the interface is not uniformly semantic: unfamiliar predicates yield no value-specific discrimination, and rare branches resist edits. The paper motivates this decomposed measurement view as a route toward, rather than a demonstration of, decomposable learning.

Critical strengths
  1. A highly insightful component-resolved measurement view. Decomposing the textual interface into four distinct requirements (Reconstruction, Interpretation, Execution, Invocation) provides a rigorous diagnostic framework. It compellingly demonstrates that a correct final answer does not certify each clause due to error cancellation, shifting the evaluation of compositional learning away from monolithic accuracy scores.

  2. Rigorous causal design and transparent boundaries. The formal proof in §3.1—showing that operation swaps cannot test predicate interpretation—is a crucial contribution. The same-record flip-vs-preserve control elegantly isolates semantic effects (+64.2 pp) from generic edit sensitivity. The authors also exhibit rare scientific integrity by explicitly bounding their claims, labeling proxy measurements, and freezing protocols before decoding.

Critical weaknesses
  1. The key negative result is un-attributable. The central boundary claim—that unfamiliar predicates yield no value-specific response (−0.35 pp)—is confounded. The authors admit (§5.3) they cannot distinguish an interface failure (failing to read the predicate) from a capability failure (lacking the arithmetic ability to compute modular residues). Without a direct-rule control, this core negative finding remains ambiguous, weakening the claim that the failure is specifically localized to the "interpretation" clause.

  2. Limited external validity due to outdated/small models and synthetic scope. The experiments only test relatively small (3B) and somewhat dated models (LLaMA-3.2, Qwen2.5). It is unclear whether the observed interface failures are fundamental properties of text interfaces or simply artifacts of low model capacity. Furthermore, the tasks are entirely synthetic, and depth-3 chains are not true compositions of independently trained modules (depth 2 is in training). The findings may not generalize to frontier models or real-world agent workflows.

Questions for authors
  1. Separating interface failure from capability failure The unfamiliar-predicate negative result (−0.35 pp) cannot distinguish between failing to interpret the written predicate and failing to compute modular arithmetic. Can you provide a "direct-rule probe" (e.g., supplying the unfamiliar predicate explicitly in the context) during rebuttal? If the model succeeds when the rule is in-context, it isolates the failure to the interface. This control is critical to my assessment of Soundness.

  2. Model scale and true compositional generalization Given the reliance on 3B models and the fact that depth-2 chains were seen during training, it is unclear if the observed limits are interface properties or just artifacts of model capacity. Do you have any preliminary results on a larger model (e.g., 7B/8B) or a strict zero-shot component composition setting? Evidence here would significantly affect my evaluation of external validity.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EZJG

Thank you for making the attribution issue explicit. We agree that the submitted −0.35-pp residue contrast does not identify whether the problem is computation or use through the text interface. New experiments and saved-output recounts below are post-submission, kept separate from the submitted results; intervals are 95% across-seed t intervals (unclipped) unless marked.

Direct-rule probes: the unfavorable results stand. The few-shot residue test qualified 0/10 adapters under a frozen gate (parse ≥.95, balanced accuracy (BA) ≥.90, sensitivity and specificity ≥.80). Supplying predicate values qualified 0/6 retrained LLaMA adapters; the Llama-3.1-8B explicit-residue probe also qualified 0/6. Familiar residues are also weaker in some explicit formats than in the native trace, so these failures neither prove arithmetic incapacity nor isolate a reading failure. We do not interpret the unqualified residue gaps.

New 8B tests constrain cross-identifier transfer.

  1. Familiar parity transfers. Written under either residue-policy identifier, it yields the prescribed branch and operation in 1,728/1,728 step-1 cases.
  2. Familiar residue transfer varies with format (updated 14 Nov). On a sealed test with host labels, Y's digit-sum predicate under Z gives branch-and-operation BA .495 [.463,.527] (5/6 seeds natively qualified), and Z's mod-5 predicate under Y .531 [.503,.560] (6/6; interval just above .5) (updated 14 Nov). Across all six seeds, behavior follows the host identifier's own trained predicate in 465/480 and 401/480 test cases; where only the written predicate is true, the written branch appears 0/120 in both directions. With donor labels, branch-and-operation BA is .905/.652 (four mod-5→Y seeds at .500 use the host's other operation); with complete rules, .963/.977 (updated 14 Nov).
  3. A value learned elsewhere shows no detected advantage. In a separate four-policy cohort, a fourth identifier learns a second mod-5 value. Each learned value, written under the identifier that did not learn it, scores no better than untrained values: −0.6 pp [−1.9,0.7] and 0.0 pp [−0.1,0.1] in the two exploratory directions (5/6 seeds qualified; no equivalence claim).
  4. Native recount (post hoc). Where the trained predicate is false, a matching unfamiliar residue activates the positive action in 1/1,728 cases. Among trained-true cases with a canonical positive action, a contradictory edit leaves it in place in 1,518/1,617 (834/836 digit sum, 684/781 mod 5; outcome-conditioned).

Transfer varies with the tested rule format; the value test (3) also kept the host's labels (updated 14 Nov). They do not establish a binding mechanism or explain why a never-trained value fails, and we keep that question open rather than claiming this resolves your critical control.

A narrower positive result for thresholds. The earlier fresh-pool Qwen threshold replication failed (1/6 qualified, three required). We then froze a comparison with identical questions and the same six worked examples in explicit and native-trace contexts (six Qwen adapters; states 200–380 and 600–900) (updated 14 Nov). Explicit minus native BA is +38.8 pp [36.1,41.5]. On separate calibration data, 3/6 qualify in the shifted stratum (gap +40.9 pp [32.1,49.7]) and only 2/6 in the original-like stratum, so an ability-versus-native-use reading is supported only for the shifted setting. The two contexts still differ in serialization and history; no internal mechanism is identified.

Larger model and composition. Llama-3.1-8B with full pair support reaches 78.5% [66.1,90.9] exact three-step accuracy and 86.1% [78.9,93.3] branch-balanced operation-swap following (six seeds × 288 cases); rare-group rates are 55.9%/61.1%. There is no 8B neutral-training control, the tasks remain synthetic, and this is not frontier-model evidence. Equivalent branch reordering (an untrained surface form) still costs 30.4 pp [21.7,39.1]. For zero-shot composition, a withheld pair stays weak at 3B (4.33% vs neutral 3.67%) and 8B (23/300, prespecified verdict “partial”); both mostly write Z's rule under X. On separate new cases, replacing only that span with the catalog rule for the emitted identifier raises accuracy from 3.7% to 96.7% at 3B (+93.0 pp [87.9,98.1] over replay) and from 2.3% to 98.3% at 8B (+96.0 pp [91.2,100.8]); an exploratory 3B extension gives repair gains on each of the five other withheld pairs (+62.0 to +92.3 pp). This is catalog-assisted repair, not independent neural modules.

The workflow effect keeps +49.9 pp [44.6,55.2] under a direction/margin-matched control (three seeds × 281 records; record bootstrap); original-threshold displacement remains unmatched.

Would the cross-identifier results and the qualified threshold comparison address your Soundness concern for this narrower diagnostic claim, with the residue result left unresolved? If not, which remaining control would be decisive?

Thank you for the additional experiments. They partially address my concerns about model scale and composition, but the key distinction between interface and capability failure remains unresolved. I will therefore maintain my original score.

Thank you for the update. We agree that the interface-versus-capability question remains open for novel residue values, and we do not claim to have resolved it. One post-submission result narrows it. With the same 8B adapters, a familiar residue predicate written as a complete rule under another identifier is followed (branch-and-operation balanced accuracy .963 and .977), but a never-trained residue value is not followed in any written form we tested, including as a complete rule or with a fresh label (.24 to .49). The failure is therefore specific to values the model never learned, not to reading written rules in general; whether it reflects reading or computing those values remains unresolved. On scale: rare-branch operation following rises from 4.5% and 2.0% (3B) to 55.9% and 61.1% at 8B, while the novel-residue probe (0/6), the withheld pair (23/300 without the catalog rule) and branch reordering (−30.4 points) persist. There is no 8B neutral-text control, so we do not claim a scaling result.

Reviewer_J9iXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether language models can use reconstructed textual policies as interfaces for composing learned skills. The authors fine-tune LLaMA-3.2-3B and Qwen2.5-3B on short sequences of conditional arithmetic policies, where each policy is associated with an opaque identifier and must be reconstructed from the model's learned parameters before execution. They evaluate generalization to held-out three-step compositions and propose four requirements for a reliable textual interface: policy reconstruction, predicate interpretation, operation execution, and continued invocation.

The main contribution is a set of controlled interventions designed to determine whether the reconstructed policy text actually influences subsequent model behavior. Operation swaps redirect action selection in 82–85% of cases on average, although performance is substantially worse on rare branches. In a separate synthetic customer-service workflow, threshold edits produce a 64.2-percentage-point decision-directed effect relative to matched decision-preserving edits. However, unfamiliar arithmetic predicate values show no meaningful value-specific discrimination, indicating limitations in semantic interpretation. Finally, a token- and position-matched comparison shows that informative policy text improves accuracy on held-out three-step compositions, with a 23.3-percentage-point advantage at a higher numeric range, although absolute accuracy remains low.

Overall, the paper provides empirical evidence that reconstructed policy text can influence subsequent computation and identifies several limitations of this interface. However, the experiments demonstrate shallow compositional generalization and local policy controllability rather than reliable composition of independently learned skills.

Critical strengths
  1. Controlled interventions provide evidence that reconstructed policy text influences model behavior: Rather than relying solely on end-to-end task accuracy, the paper uses operation swaps and matched threshold edits to test whether models respond to changes in the written policies. The customer-service experiment compares decision-flipping and decision-preserving edits on the same record, helping distinguish semantic sensitivity from generic sensitivity to text perturbations. This provides evidence that reconstructed policy text can serve as a causal control interface.

  2. The experiments reveal important distinctions between policy execution, semantic interpretation, and compositional generalization. The paper shows that models can often follow edited operations while struggling to interpret unfamiliar predicate values. The comparison between informative and semantically neutral policy text also provides evidence that policy content improves performance on held-out three-step sequences beyond simply generating additional intermediate tokens. These findings are informative because they identify specific capabilities and failure modes that need to be addressed before textual policies can support reliable decomposable learning.

Critical weaknesses
  1. Limited evidence for decomposable learning and compositional generalization. Although the paper is motivated by the goal of composing independently learned skills, the experiments do not directly evaluate this setting. The models are jointly fine-tuned on individual policies and all ordered pairs of distinct policies, and then evaluated on held-out three-step sequences. Consequently, every adjacent pair in the test sequences has already appeared during training. The results primarily demonstrate limited generalization to longer chains rather than the ability to compose independently acquired skills. This substantially limits the conclusions that can be drawn about decomposable learning, which is the central motivation of the paper.

  2. Limited generality and robustness of the empirical findings. The evaluation is restricted to synthetic arithmetic rules and a customer-service workflow, using relatively small 3B-parameter models. Even in these controlled settings, performance is inconsistent: operation-swap following deteriorates substantially on rare branches, unfamiliar predicate edits show little meaningful semantic discrimination, and absolute accuracy on held-out three-step compositions remains low. Furthermore, the advantage of informative policy text is concentrated in trace-format outputs, with no consistent benefit in the direct-answer channel. These limitations make it difficult to determine whether the proposed textual interface supports robust, generalizable composition beyond the specific training and evaluation setup.

Questions for authors
  1. Compositional generalization (additional evidence). The models are trained on all ordered pairs of distinct policies and evaluated on held-out three-step sequences. How can we distinguish genuine compositional generalization from extending previously learned pairwise transitions? Have the authors evaluated training on individual policies only, or holding out selected ordered pairs during training? Such results would materially strengthen the paper's claims about decomposable learning.

  2. Dependence on the output format (clarification). The improvement from informative policy text appears primarily in trace-format outputs, with little or no benefit for direct answers. Can the authors explain why this difference occurs and what it implies about whether the learned compositional capability generalizes beyond the supervised trace format?

  3. Predicate interpretation versus arithmetic capability (clarification and additional evidence). The unfamiliar predicate-value experiment shows almost no value-specific discrimination. Can the models correctly evaluate these unfamiliar predicates when the rules are provided explicitly, without requiring policy reconstruction? This control would help determine whether the observed failure is due to difficulties interpreting written policies or a more basic inability to evaluate the arithmetic conditions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer J9iX

Thank you. We agree: the submitted design extends chains of known pairs; it does not establish composition of independently acquired modules. New experiments and saved-output recounts below are post-submission and kept separate from the submitted tables; submitted numbers are labelled as such. Intervals are 95% across-seed t intervals (unclipped).

1. Held-out pairs. We withheld Y→X, training one shared model on the singles and the five other ordered pairs. Mastery gates passed, yet final accuracy was informative-policy D 4.33% versus neutral-span N 3.67% (six paired seeds × 50 cases; +0.67 pp [−9.06,10.39]). Trained pairs reached 99.87%/98.27%, but their arithmetic chains occur in training. The requested test therefore does not show robust unseen-pair transfer.

All 300 D outputs emit a second step and 298 name X, but only 12 write X's rule; 248 write Z's rule under X (Y→Z is Y's only trained successor). In 246 of those 248, the emitted branch, operation and next state agree with the written Z rule applied to the model's own incoming state (descriptive agreement, not mediation). The causal evidence is the intervention: on a separate pool of 50 new cases/seed, replacing only the span with the catalog rule for the model-emitted identifier raised accuracy from 3.7% to 96.7%, +93.0 pp [87.9,98.1] over replaying the model's own span (replay − uninterrupted = 0; +96.7 pp [94.5,98.8] over a length-matched neutral span). The model still selects the branch and computes the state.

At Llama-3.1-8B the held-out pair also stays weak (23/300; prespecified verdict “partial”; 264/300 again Z's rule under X), and on the same new cases the repair raises accuracy from 2.3% to 98.3% (+96.0 pp [91.2,100.8]) (updated 14 Nov). In an exploratory 3B extension, withholding each of the five other pairs in turn (six D seeds per pair) gives unassisted accuracy ≤0.7% and repair gains of +92.3, +85.7, +80.3, +90.3 and +62.0 pp (X→Y, X→Z, Y→Z, Z→X, Z→Y); every Bonferroni-over-five lower bound is above zero. This is external-catalog-assisted execution, not autonomous learned composition.

In neutral-trained N models, the catalog rule improves on replay (+32.7 pp [11.8,53.6]), but catalog minus the trained neutral card is −3.3 pp [−26.2,19.6]: no detected catalog-specific advantage over the neutral card in this control; the interval is too wide to establish equivalence.

An earlier singles-only attempt failed its primitive-mastery criterion on its evaluated state range, so it does not establish independent composition either.

2. Why trace gains exceed direct gains. By design, D and N differ only in the policy span of trace targets; direct targets are identical (byte-identical in the held-out-pair cohort) (updated 14 Nov). The direct-answer difference therefore tests transfer from trace supervision, not direct consumption of policy text. In the submitted range study, depth-3 trace D−N is +54.17/+49.31 pp (low/high); direct is +4.31/−2.78 pp (high-direct interval [−7.78,2.22]). In the held-out-pair cohort, trained-pair direct accuracy is 76.0% (D) versus 81.7% (N), −5.7 pp [−12.5,1.2]. Submitted depth-3 mixed-format accuracy remains 38.8%/31.0% (D) versus 9.5%/7.7% (N). We do not claim format-independent skill.

3. Explicit-rule capability. We ran this control and it did not certify the capability: the few-shot novel-residue probe qualified 0/10 adapters, the supplied-value follow-up 0/6, and the 8B residue probe 0/6. Familiar residues are also weak in some explicit formats, so this does not prove arithmetic incapacity; reading versus computation remains unresolved. Familiar parity transplanted under the Y/Z identifiers is followed (8B: 1,728/1,728 branch-and-operation matches). At 8B, familiar residue transplants give branch-and-operation balanced accuracy .495/.531 with host labels, .905/.652 with donor labels (four mod-5→Y seeds at .500 use the host's other operation), and .963/.977 as complete rules (digit sum→Z/mod-5→Y) (updated 14 Nov). A residue value learned elsewhere gave no detected advantage with the host's labels retained (−0.6 pp [−1.9,0.7]; no equivalence claim) (updated 14 Nov). For thresholds, with identical items and demonstrations in explicit and native contexts, six Qwen seeds show +38.8 pp [36.1,41.5]; an ability-versus-use reading holds only in the shifted stratum (3/6 qualified; original-like 2/6), and an earlier fresh-pool replication failed (1/6).

Generality. Full-support 8B reaches 78.5% [66.1,90.9] exact three-step accuracy (no 8B N control), but equivalent branch reordering, an untrained surface form with some token-count changes, loses 30.4 pp [21.7,39.1]. Tasks remain synthetic.

Does this diagnosis, with catalog-assisted repair now shown on all six ordered pairs at 3B, address your concern enough to reconsider your assessment of the contribution? If not, which remaining gap matters most?

Reviewer_J9iXOpenReview ↗

Rebuttal response

Thanks for the detailed rebuttal and additional experiments.

The held-out-pair experiments directly address my primary concern. Informative-policy accuracy drops from 99.87% on trained pairs to 4.33% on the unseen Y→X pair. The catalog-assisted repair improves accuracy from 3.7% to 96.7% on a separate evaluation pool, providing useful causal evidence that policy reconstruction is a major bottleneck.

The authors also clarify the dependence on output format. Informative-policy gains are substantial in trace outputs but smaller or absent in direct answers (+4.31/−2.78 points).

The explicit-rule controls address my third question experimentally. It does not establish whether unfamiliar-predicate failures arise from interpretation or arithmetic computation. Furthermore, the 30.4-percentage-point degradation under equivalent branch reordering reinforces concerns about robustness.

These controlled interventions, particularly the matched threshold edits and catalog-assisted repair, are methodologically meaningful and provide useful causal evidence beyond end-to-end accuracy.

My remaining concern is that the broader significance of their findings remains insufficiently established. The experiments identify reconstruction failures, format dependence, and representation sensitivity, but it is unclear how much these observations advance our understanding of decomposable learning beyond the specific synthetic setting.

I appreciate the authors' commitment to narrow their claims and acknowledge the negative results. While the rebuttal strengthens the paper's diagnostic contribution, my concerns about its significance and generality remain. I therefore maintain my original rating.

Thank you for the follow-up and for acknowledging the held-out-pair experiments. On significance: the framework turns "the model fails on an unseen combination" into a located, measurable failure. On the withheld pair, all 300 outputs continue to a second step and 298 name the requested identifier, so invocation and naming hold; in 246 of the 248 cases where the model writes Z's rule under X, its branch, operation and next state agree with that written rule; what fails is recalling the right rule, and supplying it restores accuracy from 3.7% to 96.7%. End-to-end accuracy alone (4.33%) cannot tell this apart from a model that cannot execute the step at all. We think this is the general point: a textual interface can be checked clause by clause (reconstruction, interpretation, execution, continued invocation), and each clause can fail on its own. We do not claim that the specific rates transfer beyond these synthetic tasks.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: diagnostic scope, held-out-pair repair, and limits

Thank you for the careful reviews. We agree that the submitted setting—one shared model trained on single policies and all ordered distinct-policy pairs—does not establish composition of independently acquired modules. Our contribution is a diagnostic framework separating policy reconstruction, interpretation, execution and continued invocation. The new controls add a constructive repair and sharper limits.

New experiments and saved-output recounts below are post-submission and reported separately from submitted measurements. Intervals are 95% across-training-seed t intervals (unclipped) unless marked; “pp” means percentage points.

Unseen composition: failure and controlled repair. Withholding Y→X from training gave informative-policy D only 4.33% final accuracy versus neutral-span N 3.67% (six paired seeds × 50 cases; +0.67 pp [−9.06,10.39]), although mastery checks passed. All 300 D responses continued to a second step, but 248 wrote Z's rule under the label X. On a separate pool of 50 new Y→X cases/seed, replacing only the step-2 rule span with the catalog entry for the identifier the model itself emitted raised accuracy from 3.7% to 96.7% (+93.0 pp [87.9,98.1] over replaying its own span; +96.7 pp [94.5,98.8] over a token-length-matched neutral span). No branch choice, state or answer is supplied.

At Llama-3.1-8B the unassisted held-out result is also weak (23/300; prespecified verdict “partial”; 264/300 again Z's rule under X), and on the same new cases the repair raises accuracy from 2.3% to 98.3% (+96.0 pp [91.2,100.8]). In an exploratory 3B extension, withholding each of the five other ordered pairs in turn (six D seeds per pair) gives unassisted accuracy ≤0.7% and repair gains of +92.3 (X→Y), +85.7 (X→Z), +80.3 (Y→Z), +90.3 (Z→X) and +62.0 pp (Z→Y); all Bonferroni-over-five lower bounds are above zero (smallest +55.4 pp). This is catalog-assisted repair on all six ordered pairs at 3B and on Y→X at 8B, not autonomous composition.

Predicate transfer (updated 14 Nov). Explicit novel-residue probes qualified 0/10, 0/6 and 0/6 adapters, so residue computation versus reading stays unresolved. At 8B, familiar residue transplants give branch-and-operation balanced accuracy .495/.531 with host labels, .905/.652 with donor labels (four mod-5→Y seeds at .500 use the host's other operation), and .963/.977 as complete rules (digit sum→Z/mod-5→Y; 5/6 and 6/6 seeds natively qualified) (updated 14 Nov). In a separate four-policy 8B cohort, a residue value learned elsewhere gave no detected advantage over untrained values with the host's labels retained (−0.6 pp [−1.9,0.7] and 0.0 pp [−0.1,0.1]; 5/6 seeds qualified; no equivalence claim) (updated 14 Nov). Familiar parity also transfers across identifiers (1,728/1,728 branch-and-operation matches) (updated 14 Nov). These tests constrain behavior; they do not explain why a novel residue value fails.

For thresholds, with the same items and the same six worked examples in explicit and native-trace contexts, six Qwen seeds show a 38.8-pp [36.1,41.5] balanced-accuracy gap; an ability-versus-use reading holds only in the shifted stratum (3/6 qualified; original-like 2/6), and serialization and history still differ. The earlier fresh-pool threshold replication failed (1/6).

Scale. Full-pair-support Llama-3.1-8B reaches 78.5% [66.1,90.9] exact three-step trajectories (six seeds × 288 cases), but reordering equivalent labeled branches—an untrained surface form in which token counts change for some rules—costs 30.4 pp [21.7,39.1] in branch-and-operation accuracy. There is no 8B neutral-training control and no causal scaling claim; tasks remain synthetic.

Revision. We will foreground the diagnostic scope and the held-out-pair failure in the abstract, introduction and conclusion; place absolute accuracy, per-branch outcomes and trace/direct differences beside relative gains; add separately labeled post-submission sections on held-out-pair repair and on predicate transfer and scale; and remove rarity-alone causal language, since branch-balanced training did not support its primaries. The workflow section keeps its churn, threshold-displacement and oracle-location limitations. This paper shares the synthetic arithmetic task, the lineage of its data generator and the LLaMA training recipe with the anonymously cited concurrent manuscript; the measurements reported here are distinct, and the revision will state this. Reproducibility correction: the statement says the supplementary contains data/ and scripts/verify_statistics.py, but the submitted ZIP does not. We will correct the statement and include these files in the revised supplementary. The script is an aggregate audit of the submitted statistics, not a full reproduction package. No new result overwrites a submitted measurement or a failed protocol.

AuthorsOpenReview ↗

Summary of the revision

Summary of the revision (uploaded 19 Nov)

We have uploaded a revised PDF and supplementary material. All changes are shown in blue; results obtained after submission are marked † and are never pooled with submitted measurements. The main changes:

  1. Scope. The abstract, introduction and conclusion now state that one shared model is trained on every single policy and every ordered pair of policies, so the submitted design extends chains of known pairs rather than composing independently learned modules (J9iX, EZJG, ZHuQ).
  2. Absolute numbers first. The abstract reports absolute depth-3 accuracy next to the D–N gain; Table 2 adds the two rare branches (Y:s, Z:d); §5.4 adds the trace versus direct-answer breakdown (ZHuQ, J9iX).
  3. Rare branches. The sentence naming rarity as the cause is replaced by the branch-balanced training result, which did not support it (§5.1, ZHuQ).
  4. Workflow limits. §5.2 states that the free-run figure is a coverage-weighted edit effect, not an accuracy gain, and adds the churn replication, the direction/margin-matched control and the natural-history repair (ZHuQ).
  5. New §5.6 †: the held-out-pair test, the wrong-rule recall it reveals, and the catalog-assisted repair (J9iX, EZJG, ZHuQ).
  6. New §5.7 †: predicate transfer and the Llama-3.1-8B results, including the residue probes that remain unresolved (EZJG, J9iX, ZHuQ).
  7. Appendix K †: protocols, gates and per-seed details for the post-submission experiments.
  8. Related work. §2 states that this paper shares the arithmetic task, the generator lineage and the LLaMA recipe with the anonymously cited concurrent manuscript; the measurements are distinct.
  9. Reproducibility. The statement now says that the submitted archive lacked data/ and scripts/verify_statistics.py. The revised supplementary adds them, with internal names removed and no computation changed; it is an aggregate audit, not a full reproduction package.

The main text still ends on page 10.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.