acceptodds
Under review as a conference paper at ICLR 2027

Adapting, Fast and Slow: On Few-Shot Transportability of Compositions

Abstract

Rules learned from source sequences can be composed to learn new computations from few target samples. We study the conditions and sample requirements for this form of compositional generalization in finite sequential causal models without unobserved confounding. Causal structure and knowledge of shared mechanisms identify local conditional distributions that can be reused across domains. We give a sufficient condition under which their estimates, called gates, can be composed by Circuit-TR to estimate the target concept—the output conditional given the inputs—without target samples. When the target composition and its source correspondences are unknown, we prove that Circuit-AD competes with this oracle by minimizing empirical log loss with a circuit-size penalty. The excess-risk bound separates source estimation error from composition selection; the selection cost decreases with target sample size and adapts to the comparator's size without requiring it in advance. Different compositions can compute the same noiseless function yet incur different risks when their gates are noisy. Neural-AD learns compositions of frozen source gates, allowing us to examine these distinctions experimentally. The results show how the available source operations affect adaptation and how process supervision reduces the samples needed to learn a composition.

Primary area:
causal reasoning
Keywords:
Compositional Generalization, Causal Transportability, Domain Adaptation, Sequence Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_i6rmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how operations learned from source examples can be combined to solve new tasks with little target data. Circuit-TR assumes the correct composition is known, while Circuit-AD learns it from target examples. The theory separates errors in learned operations from errors in choosing their composition. Neural-AD provides a practical search method. Experiments test synthetic computations, GCD, electricity-data aggregates, and supplied compositions of operations extracted from language models.

Critical strengths
  1. The paper separates errors in learned operations from errors in choosing how to combine them. It also shows why operations that produce the same answer without noise can behave differently with noise.

  2. Testing every possible circuit in small settings provides a strong reference for the search method. Comparing learning with and without intermediate steps helps explain when learning fails.

Critical weaknesses
  1. Very Limited novelty over earlier methods. Earlier work like Modular meta-learning and Neural Relational Inference with Fast Modular Meta-learning already learns reusable modules and searches for ways to combine them, including learned search proposals. The theory brings these ideas together, but the neural guarantee assumes good search results without showing that the method achieves them.

  2. The comparisons leave the method’s benefit unclear. The paper lacks a comparison with earlier search methods using the same learned operations and computing budget. Several electricity baselines perform worse than uniform guessing under log loss. The method also struggles with a nine-position target despite an available accurate solution.

Questions for authors

1.How does Neural-AD compare with earlier modular search methods using the same frozen operations, objective, and computing budget?

2.In the small enumerable settings, what are the actual optimization residual and Jensen gap in Theorem C.5 for the mixture predictor?

3.Why do several electricity baselines perform worse than uniform prediction under log loss? Does improving their calibration or optimization change the conclusions?

1 comment

We appreciate Reviewer i6rm’s suggestions and feedback. We invite them to view the updated manuscript, as we address their concerns below.

Reviewer i6rm: “Very Limited novelty over earlier methods.” The Reviewer points to modular meta-learning and its use of learned search proposals.

We made this connection explicitly in the submission.

lines 3373–3374: “Alet et al. (2018) provide a particularly close adaptation procedure: learn neural modules across tasks, freeze them, and select a composition by empirical loss on a new task.”

Our question is what makes two sequence tasks related enough that pretraining on one can make the other learnable from few examples. Tasks with different answers can share the mechanisms that compute individual steps. We show when next-value prediction identifies those operations and how many new examples suffice to approach the predictive risk of an oracle given their arrangement. The transformer gives us one way to test that claim, and it does not refute possibility of other algorithms, generic or specialized alike, to demonstrate this capacity. A central message of the paper is precisely in this regards: adaptability from task A to task B is a quality of the data-generation mechanisms underlying the task A and B, not the quality of a particular learning algorithm. We show, as a proof-of-concept, it is plausible that optimizing a next-token prediction objective on task A allows a structure-agnostic model, such as the one introduced in this paper, to adapt faster to a domain B, as a mere side-effect of the training objective. even when the true intermediate execution trace that hints at the true composition is hidden to the learner.

Reviewer i6rm: “How does Neural-AD compare with earlier modular search methods using the same frozen operations, objective, and computing budget?”

Before responding, we would like to emphasize, once again, that the main message of the paper is not about a particular architectural choice and training procedure. We believe it would be clear from the exposition, but your feedback indicated that we need to make the distinction more clear to avoid misconceptions like this. With the same frozen gates, starting circuits and trace scores, modular annealing and our learned proposal selected identical circuits from 16 traces on each of two tasks. Whole-trace accuracy was 96.8% on the first task and 96.9% on the second. Annealing used at most as many nominal gate evaluations (revised Appendix K.5). With 1,024 pretraining pairs per task and 32 new examples, BounceGrad and Fast Modular learned libraries that gave 79.3% and 61.5% trace accuracy, ahead of Neural-AD’s 42.7%.

Reviewer i6rm: “[...] what are the actual optimization residual and Jensen gap in Theorem C.5 for the mixture predictor?”

We did not measure either quantity for the submitted result. Submitted Table 6 reports the fitting gap of one decoded circuit. The revised version returns a single circuit, for which the Jensen gap is zero by construction. With full traces, both searches attained the exact empirical minimum in all 120 unary fits (revised Appendix K.13).

Reviewer i6rm: “The method also struggles with a nine-position target despite an available accurate solution.”

Correct. Through 500 answers the penalized objective favored the one-gate circuit. At 1,000 and 2,000 answers it favored the supplied circuit, and search missed it (submitted lines 2725–2730). In a separate revised experiment on three-input range, changing the annealing schedule at the same proposal budget raised 16-answer accuracy from 59.0% to 99.9% (Appendix K.4).

Reviewer i6rm: “Why do several electricity baselines perform worse than uniform prediction under log loss? Does improving their calibration or optimization change the conclusions?”

The learner must transfer arithmetic from synthetic traces to real electricity readings. On largest adjacent change, a fresh baseline trained on 100 examples assigned the true answer probability below 0.00011 on 81% of held-out windows. Uniform assigns it 0.01. Averaging each of three validation-selected predictors with uniform brought their losses to 4.39–4.50 nats, below uniform’s 4.61. On all 20,000 test windows, a smoothed answer histogram scored 4.34 nats. The circuit learner scored 0.067 (submitted Table 10). Better calibration reduces the margin. The learned operations still transfer.

Reviewer_xpmXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed three modules:

  1. Circuit-TR: Assuming that the underlying computational mechanisms are shared between old and new tasks, the authors prove that previously learned operations can be directly reused and derive an upper bound on the accumulated computational errors. This serves as a reference case where the correct computational procedure is known.

  2. Circuit-AD: When the correct computational procedure is unknown, the authors prove that a composition selected using a small number of target-task samples can achieve a prediction error close to that of the reference case above. As the number of samples increases, the upper bound on the additional error introduced by composition selection decreases.

  3. Neural-AD: The authors use a Transformer to search for compositions of computational operations and experimentally evaluate whether its performance can approach that of exact search.

The authors' main new contribution is a unified theoretical guarantee that accounts for both the probabilistic estimation errors of previously learned modules and the statistical errors from selecting their compositions for new tasks.

Critical strengths
  1. The paper presents a coherent and complete narrative. It is easy to follow and has no obvious technical issues.
  2. Circuit-AD and Neural-AD use the same gates and candidate search space in the short-circuit experiments, making the comparison reasonably convincing. The authors also transparently report cases where the search fails.
Critical weaknesses
  1. I basically consider this paper to be a useful combination and specialization of existing theoretical results, with limited theoretical novelty. The authors assume that the underlying mechanisms are shared across tasks. Their proofs mainly rely on classical statistical tools. Furthermore, I can't see if Neural-AD provides any new guarantees on search efficiency. See the Questions section for details.

  2. Experimental results show that Neural-AD fails to find good circuits for longer circuits, and there is no theoretical guarantee that Neural-AD can efficiently find good circuits.

  3. The scope of experimental validation is still somewhat limited. Section 6 and Appendix G.7 mainly focus on artificial computational tasks. The target task in the electricity dataset is defined by the authors. The Qwen experiment provides the target circuit. The paper has not yet demonstrated effectiveness on real-world tasks with unknown structures.

  4. Figure 3 is based on a single source model training run, with repeated sampling of target data. Although the paper acknowledges this limitation in Appendix J, this limitation makes it hard for readers to know whether the conclusions would remain stable after retraining the source model.

Questions for authors
  1. Circuit-TR builds on existing causal transfer theory and KL-divergence-based error analysis, while Circuit-AD relies on classical complexity penalties and concentration inequalities. Compared with prior work on causal transfer theory, as well as the complexity and sample complexity analyses of Playgol and Barron & Cover, what important new results does this paper establish that could not be obtained from existing theories?

  2. When using the same gate library and candidate search space, how do Circuit-AD and Neural-AD compare in terms of runtime, number of search steps, and prediction error? (The authors acknowledge that exhaustive search is computationally expensive. However, the experiments do not seem to sufficiently demonstrate the practical computational benefits under comparable conditions. )

  3. Compared with directly assuming that the conditional distributions of the source and target modules are identical, what additional substantive conclusions do the SCM formulation and causal correspondence provide in the paper's theorems?

1 comment

We appreciate Reviewer xpmX’s careful reading of our work, and we invite them to view the updated manuscript. We address the Reviewer's concerns below.

Reviewer xpmX: “Their proofs mainly rely on classical statistical tools.” The reviewer asks what we add to causal transfer theory, Playgol, and Barron & Cover.

Yes. Our concentration and complexity-penalty arguments use established tools, including the principle of Barron and Cover. SCAN (Lake and Baroni, 2018), for example, tests zero-shot generalization to held-out compositions. Playgol shows how learning predicates can shorten programs and improve a sample bound. Our question is when next-token prediction on the execution traces of familiar tasks can make an unseen task learnable from a few examples, and which relationships between the tasks determine how many examples suffice. The cited results do not give that characterization.

We use causal transportability to determine what an oracle could predict if it knew the new wiring and the shared mechanisms. We then ask how many examples can take the place of that knowledge. Revised Theorem 3.1 shows when pretraining traces identify the gates without operation or argument labels. Each full trace of the new task tests local choices of gate and arguments (Theorem 3.3). With answers alone, a learner can compete with any composition of the learned gates, paying for its predictive error and description length (Theorem 5.2).

Why use a causal model if we could assume equal tables directly? In submitted Example D.1, two tasks share the outcome mechanism and the input distribution, yet confounding permits P⋆(Y=1∣X=0)P^\star(Y=1\mid X=0) to be 0.26 or 0.74. The SCM tells us when mechanism sharing implies table equality. Here, even knowing the pretraining distribution exactly leaves the answer undetermined.

Reviewer xpmX: “[...] how do Circuit-AD and Neural-AD compare in terms of runtime, number of search steps, and prediction error?”

In the revised nine-step unary study, exact selection takes 180 local scores because full-trace likelihood separates by position. The learned proposal ran 1,000 updates with 16 sampled circuits per update. Both neural search procedures (the learned proposal and annealing) attained the exact empirical minimum and matched the supplied circuit’s 90.8% whole-trace accuracy (Appendices K.5, K.11 and K.13). With 16 answers alone on three-input range, the selected circuits averaged 39.5% accuracy over all inputs, against 99.99% for a supplied circuit using the same gates (Appendix K.4). Wall-clock speedup remains unmeasured.

Reviewer xpmX: “The paper has not yet demonstrated effectiveness on real-world tasks with unknown structures.”

The natural-task demonstration remains open. In revised Appendix K.3, we withheld the wiring on eight controlled pretraining tasks. The learner received ordered traces, opaque task IDs and an input/output contract. At 1,024 pairs per task, each of three independently fitted libraries recovered the correct parent at all 16 computed positions. The controlled setting lets us check that claim directly.

Reviewer xpmX: “Figure 3 is based on a single source model training run, with repeated sampling of target data.”

Correct. Those error bars are conditional on one pretraining fit. In a separate unary study, we repeated pretraining on two fresh datasets with independent fitting seeds. Given 32 examples per new task, the two libraries achieved 96.78% and 95.75% mean trace accuracy across two tasks (revised Table 11).

Reviewer_detvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is very clearly AI-generated and the exposition is extremely hard to follow. I have made multiple passes of abstract and introduction at this point and the problem statement and contributions are still not very clear to me. Terms and concepts are used without defining them clearly. For e.g. in the following line: "Causal transportability asks whether available distributions and knowledge of shared and changing mechanisms determine a target quantity (Pearl & Bareinboim, 2011; Correa & Bareinboim, 2019).", the notion of target quantity and its relevance is never defined in the paper, and what constitutes the shared and changing mechanisms is also not very clear.

The authors also clearly mention in the AI Use Statement the following:

""" In this work, we used ChatGPT, Codex, Claude, and Cursor for the following purposes:

• Ideation, writing, choice of notation, and structuring the paper.

• Literature overview and references.

• Drafting and checking mathematical arguments, including the conditional-estimation and selection proofs.

• Developing the codebase and preparing figures.

• Interpreting experimental results and preparing the reproducibility and implementation descriptions in Appendix G.1.

The authors are responsible for the final text, claims, proofs, code, and results. """

Given this I am not sure what was the contribution of authors, as paper writing, literature review, mathematical arguments and proofs, and even interpreting the experimental results was done using AI tools. I am not sure if it is the best use of my time reviewing this paper, which is also completely unreadable.

Critical strengths

NA

Critical weaknesses
  1. Paper is completely unreadable.
  2. It's not at all clear what was the contribution of the authors in the whole project
Questions for authors

NA

1 comment

We are making this rebuttal for future readers.

Reviewer detv: "[...], the notion of target quantity and its relevance is never defined in the paper, [...]"

The submission described the target quantity in the abstract,

lines 16–18: "[...] to estimate the target concept—the output conditional given the inputs—without target samples"

and defined it on page 3 of the manuscript.

line 110: "Their conditional table P⋆(y∣x)P^\star(y\mid x) is the target concept."

Reviewer detv: "[...] and what constitutes the shared and changing mechanisms is also not very clear."

Check out pages 3–4 for a formal definition:

lines 161–163: "Definition 3.2 (Discrepancy oracle). The discrepancy oracle Δ(i,i′)\Delta(i,i') is zero when the supplied information certifies that target position ii and source position i′i' have the same arity, structural function of the ordered arguments, and exogenous noise distribution. A value of one imposes no equality requirement."

Reviewer detv: "I have made multiple passes of abstract and introduction at this point and the problem statement and contributions are still not very clear to me."

It is common practice to contextualize the problem broadly and without heavy notation in as little as one-and-a-half pages (which include the title, abstract, and introduction) before a formal problem statement that needs proper notation. But check out the second line of the abstract:

line 13: "We study the conditions and sample requirements for this form of compositional generalization in finite sequential causal models without unobserved confounding."

If the above didn't ring a bell, it implies that Reviewer detv was unfortunately assigned to a topic outside of their interests. Nonetheless, a formal definition of circuit-transportability was available at page 4 of the manuscript:

lines 167–169: "Definition 3.3 (Circuit transportability). The query P⋆(vT⋆⋆∣v1:M⋆)P^\star(v^\star_{T^\star}\mid v^\star_{1:M}) is circuit-transportable from PP given Δ,G,G⋆\Delta, G, G^\star if it has the same value for every source–target SCM pair whose source model entails PP and that satisfies those diagrams and mechanism-sharing constraints."

Finally, Reviewer detv posted a copy of our AI use statement into their paper summary section, and remarked: "Given this I am not sure what was the contribution of authors, as paper writing, literature review, mathematical arguments and proofs, and even interpreting the experimental results was done using AI tools. I am not sure if it is the best use of my time reviewing this paper, which is also completely unreadable."

The AI-use disclosure was required by the guidelines, and we deliberately disclosed it in detail for every stage of the process that AI ever touched, in the past two years since the start of the project. It is called a “disclosure” for a reason, and it was not a hidden subtext discovered by Reviewer detv.

We did not appreciate a dismissive review.

P.S.

Reviewer detv: "Paper is completely unreadable."

The other two reviewers read the same manuscript. Check out what they wrote in the summary section:

Reviewer xpmX, summary: "The authors' main new contribution is a unified theoretical guarantee that accounts for both the probabilistic estimation errors of previously learned modules and the statistical errors from selecting their compositions for new tasks."

Reviewer i6rm, summary: "The paper studies how operations learned from source examples can be combined to solve new tasks with little target data. Circuit-TR assumes the correct composition is known, while Circuit-AD learns it from target examples. The theory separates errors in learned operations from errors in choosing their composition."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.