acceptodds
Withdrawn from ICLR 2027

In-Context Learning of Anonymous Matrix Operators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We study how to infer a shared distribution transformation from unpaired input and output samples and apply it to a new input distribution. The samples form matrices with unordered rows and unlabeled but aligned columns. The new output distribution can be identifiable even when the operator itself is not. For shared Markov kernels, we characterize input backgrounds that uniformly identify the query and derive a prediction lower bound when contexts are indistinguishable. For affine-Gaussian responses, we establish constructive recovery and stability conditions. We also derive the Bayes response law under squared MMD and analyze conditional flow matching with finite pretraining. ACEMO-ICL implements this approach with a conditional generator that respects the matrix symmetries. Small-budget experiments show that it uses demonstrations and benefits from informative backgrounds.

Primary area:
learning theory
Keywords:
In-context learning, Identifiability, Distribution-to-distribution learning, Conditional generative modeling, Permutation equivariance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZmMzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies response-distribution prediction from unpaired demonstrations of a shared unknown operator. It characterizes population query identification, derives affine-Gaussian recovery conditions, and proposes ACEMO-ICL, a conditional flow model. Synthetic results support context use; real-data transfer is inconsistent and representation-sensitive.

Critical strengths

The distinction between identifying a query response and identifying the full operator is useful. The population characterization makes this distinction precise. Context interventions demonstrate that the model uses input–response relationships. Negative results and limitations are reported transparently.

Critical weaknesses

My main concern is that the strongest theoretical results are only loosely connected to what ACEMO actually learns. The paper characterizes when population evidence identifies a query response, but does not establish when the trained predictor successfully extracts that information. The finite-observation span illustration assumes known input laws and coefficients, while none of the sampled affine conditions falls within the sufficient stability radius. The neural risk bounds leave approximation, optimization, and regularity terms uncontrolled for the implemented model. These results may be mathematically sound, but they do not explain when ACEMO succeeds from finite demonstrations or why its architecture should exploit identifying information better than a standard conditional flow.

The generality of the framework also depends on substantial modeling assumptions. Backgrounds must share one response mechanism, although changing the biological or environmental background may itself change that mechanism. The real-data examples do not establish when this assumption is appropriate. Moreover, although a Markov kernel can describe nonlinear individual responses, its action on the input distribution is linear. Population-dependent responses, interactions between individuals, and normalized reweighting therefore fall outside the general identification theorem. This limits how broadly the theoretical conclusions can explain the response-transfer problems motivating the work.

My other main concern is that the practical benefit of learning the operator remains unconvincing. On the original Kang18 representations, ACEMO performs substantially worse than simply reusing support responses (Table 7). Its improvement follows an exploratory rotation investigated after the original failure, and the rotated model wins on only 4/8 and 2/8 donor folds. On Bodenmiller, it does not improve the mean over support-response resampling. The authors disclose these outcomes appropriately, but the results still leave the central practical question unresolved: when does learning this conditional operator provide a reliable advantage over a much simpler predictor?

The architectural evidence does not resolve that question either. Comparisons with set flow differ in auxiliary source supervision and computation, so the gains cannot be attributed cleanly to the proposed design. The matched token-compression comparison yields only about a 0.34% improvement. Together, these results provide stronger evidence that context is useful than that the particular ACEMO architecture contributes a substantial methodological advance.

Finally, the prediction target limits the scientific interpretation. Independent samples from the posterior predictive marginal do not represent the joint posterior in which an entire output batch shares one unknown mechanism. This is appropriate for the chosen marginal MMD objective, but it does not support mechanism-level uncertainty or coherent batch-level posterior prediction. The paper acknowledges this distinction; its implications for the intended applications remain important.

Questions for authors

Please address the concerns raised in Critical Weaknesses, particularly the theory–learning connection and the evidence for reliable practical transfer.

Reviewer_eiW7OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an in-context learning setting where each demonstration contains an unordered set of input samples and an independently sampled set of output samples. The rows are not paired, while the columns are aligned across demonstrations. The goal is to predict the output distribution for a new query input distribution. The paper provides theoretical results describing when the query response can be identified from the demonstrations, including a general result for shared Markov kernels and a more constructive analysis for affine Gaussian mechanisms. It also proposes ACEMO-ICL, a conditional flow model designed to respect the permutation symmetries of the problem. The experiments include synthetic tasks, ablations, out-of-distribution tests, and two observed-data examples.

Critical strengths
  1. The problem setting appears different from standard supervised learning. The lack of row-level input–output pairing makes the task nontrivial, and the distinction between identifying the full operator and identifying only the requested query response is useful.
  2. The paper combines comprehensive theory and experiments.
Critical weaknesses
  1. The paper is difficult to enter for readers who are not already familiar with the mathematical background. Several concepts and symbols are introduced quickly, and the intuitive meaning of the main theorems is sometimes harder to follow than the formal statement. A small running example used throughout Sections 3–5 would improve accessibility.
  2. The connection between the theory and the neural model could be signposted more explicitly. I understood the theory as describing when the query is identifiable, but I was less certain about which parts of ACEMO-ICL directly correspond to the theoretical quantities. A short paragraph or diagram linking the assumptions, architecture, and experiments would make the overall story easier to follow.
Questions for authors
  1. Could the authors add a simple one-dimensional or two-dimensional example before Theorem 3.1 to illustrate what it means for a query distribution to lie inside or outside the span of the demonstrations?
  2. What real-world application do the authors consider the most natural motivation for this anonymous and unpaired matrix setting?
  3. Could the authors provide short pseudocode for the training and inference procedure of ACEMO-ICL? This may help readers understand which quantities are visible at each stage.
Reviewer_BZA6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prediction from unpaired input and output samples. Each task uses one shared transformation across several input distributions. The main question is whether these examples determine the response to a new input distribution. The paper separates this question from recovery of the complete transformation.

For general Markov kernels, the paper gives an exact condition for a uniform identification guarantee. It also gives recovery conditions for affine-Gaussian transformations. The ACEMO-ICL model uses these unpaired examples to generate a new response. Experiments include both synthetic tasks and two real-world biological datasets.

Critical strengths

The research question is valuable. More samples cannot resolve an ambiguity that the input distributions preserve. The paper makes this distinction precise. This result can help researchers select informative demonstrations for learning across distributions.

The theory matches the research problem closely. The span condition identifies when the demonstrations must determine the query response. The approximate bound separates missing information from amplification of observation error. The affine analysis gives a more concrete example of these effects.

The experiments are comprehensive. Incorrect operators and reassigned response sets reduce prediction quality. These experiments support the value of demonstration content and pair membership. The paper also reports failed recovery conditions and weak real-data results, which is still appreciated and help readers understand the scope of the proposed method.

Critical weaknesses

It is unclear how much the model design helps. The model performs better than set conditional flow in Table 2. However, it also gets extra information during training and uses more training time. When the comparison controls for these differences, the token design gives only a small improvement. The results therefore give stronger support for the new problem formulation than for the model design.

The benefit on real data is still unclear. On Kang18, the model performs poorly with the original data representation. After the authors rotate the data representation, the average result improves, but the benefit is not consistent across people’s cell samples. On Bodenmiller, simply resampling the observed support responses gives a lower average error. The theory remains useful, but these results do not yet show a clear practical advantage.

Questions for authors

How much of the gain over set conditional flow remains with the same source supervision and training budget? Does the Kang18 improvement persist across several rotations chosen without query-response data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.