acceptodds
Under review as a conference paper at ICLR 2027

Six Requirements for Inference-Time Self-Iterating Models: Representation, Uncertainty, and Local Credit

Abstract

Inference-time self-iteration lets a model run an iterative learner on its context before it answers. For multi-branch in-context relations—how different two items are along an attribute the model was never trained on, or compressive phase retrieval—a lifted inner loop (per-example global inversion, multipliers, local least squares) provably reaches what test-time training's gradient update, fixed-feature closed-form heads and additional layers cannot. Whether a pretrained model can host such a loop depends on its representation. We prove six requirements. Two concern the representation and hold for every learner: the task must be decodable from the base, which sets an explicit error floor floor(ν) in terms of the base's undecodable fraction ν (R1), and it must be identifiable from the context, which ties the task dimension to the context size and calls for posterior mixtures near the threshold (R2). Two concern uncertainty: inversion and readout must be consistent with the base's residual noise (R3), and in-context noise estimates must be corrected for degrees of freedom (R4). Two concern local credit: the multiplier leak must vanish on exactly decodable bases and grow with the noise (R5), and the knobs that control branch changes must be selected, not trained by backpropagation (R6). R1 and R2 give measurable diagnostics of a base. On word vectors (GloVe), a molecular graph network (GINE) and an image code (PCA), which satisfy them, the matched self-iterating learner beats gradient test-time training, quadratic closed-form heads and Transformers, with a second-seed replication and paired tests; on a base that violates decodability (MiniLM word vectors) every learner, including an oracle probe, stays above the proven floor.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
inference-time self-iteration, pretrained model adaptation, test-time learning, writable memory, representation learning, uncertainty calibration, local credit assignment, augmented Lagrangian methods

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BQtZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes six requirements for inference-time self-iteration and introduces a solver combining local inversion, multiplier updates, and least squares. The overall idea is interesting, and the controlled experiments are fairly systematic. However, I do not think the current theory and experiments support the paper’s strong claims.

Critical strengths

Discussing representation quality, noise handling, and inner-loop optimization within one framework is a useful perspective. The controlled experiments also help explain the individual design choices.

Critical weaknesses
  1. Calling these “six requirements” is too strong. R1 and R2 describe information limits under specific statistical assumptions, while R3–R6 are largely tied to the proposed solver. For example, the optimal-regularization argument behind R5 assumes a fixed correct branch assignment. This does not establish a general requirement for self-iterating models.

  2. The presentation feels rushed and insufficiently polished. Assumptions, theoretical conclusions, and implementation choices are often packed together without clearly explaining their connections. The authors also disclose using AI coding agents to produce the plotting scripts. And the presentation and unresolved gaps leave me feeling that the submission needs substantial further development.

  3. Important baselines and compute comparisons are missing. The problem is closely related to phase retrieval, yet the experiments do not directly compare against AMP, Wirtinger flow, or classical HIO/DR methods discussed in the paper. The vision experiments also use eight chains for the proposed method while all baselines are single models. Without matching computational budgets, the source of the improvement remains unclear.

  4. There are several gaps between the theory and its practical application. Poor linear-probe performance does not establish that a representation contains no information accessible to nonlinear learners. The noise correction derived for a fixed linear smoother is not shown to remain unbiased after adaptive branch selection. Likewise, difficulties with gradient-based optimization do not establish that derivative-free selection is necessary. Finally, although the experiments cover three domains, they largely share the same specialized multi-branch inverse-problem structure, so the broader applicability remains uncertain.

Questions for authors

See the weaknesses above.

Reviewer_qfJDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Methods that use a small dataset of examples provided in context at test time have been shown to significantly improve inference. This work provides theoretical bounds on the error that can be achieved on a particular family of test time inference problems using a fixed feature encoder, and empirically validates these bounds. The authors provide a lifted inner loop algorithm that achieves better performance than its fixed feature representation counterparts, as well as requirements for similar algorithms based on theoretical results.

Critical strengths
  • The theoretical results appear to be novel, and many of them are interesting and generalizable, such as the bounds on fixed feature models.
  • Extensive empirical validation is done using a variety of different models in different application domains.
Critical weaknesses
  • [W1] The writing is very unclear, and the paper is difficult to follow. Several sections are incredibly dense and allude to concepts that can’t be understood without further context. For instance, the introduction’s description of the six requirements mentions several concepts that are meaningless at that point in the paper. Theoretical results are very densely packed together with little explanation, often times making them difficult to interpret. The formatting is poor: the paper does not use the ICLR style, points meant for signposting are inserted into the text without any additional formatting leading to some confusion (lines 58-68 have examples of this), algorithm 1 is poorly formatted. Finally, the setting (i.e. what the inputs, keys, and labels are) is at times not clear (see questions).
  • [W2] The majority of the paper’s contributions are limited in scope. The paper focusses on its lifted inner loop algorithm, which appears to be an adaptation of a predictive-coding based approach from Seely & Gould (2026). This style of model is not very broadly used, and so while the theoretical results regarding it are interesting, they are not of relevance to most researching test time training. Furthermore, while the work motivates using approaches that dynamically learn a feature representation from the context over fixed feature approaches (theorem 1), focussing on this lifted inner loop algorithm as opposed to other approaches that dynamically learn features is not well motivated.
Questions for authors
  • [Q1] What is being provided in context to any of these algorithms? Is it the difference between pairs of key values? Is it the key itself? What is the label? The inner product with the label vector u? The difference between these inner products?
  • [Q2] In table 2, I do not understand what the overlap column is. What is an end-to-end TTT layer? Are these models able to do feature learning?
Reviewer_jUaxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is shockingly poorly written, and it is impossible as a human reader to understand what they are doing. I detail my concerns in the weaknesses section below

Critical strengths

None.

Critical weaknesses

The paper in its form is just incredibly hard to read and understand. Even after careful reading, it is not clear to me the problem they are trying to solve or the motivation behind it. Let me make it clear with examples.

Reading the abstract, right after the first line, the abstract says "For multi-branch in-context relations–how different two items are along an attribute the model was never trained on, or compressive phase retrieval—a lifted inner loop (per-example global inversion, multipliers, local least squares) provably reaches what test-time training's gradient update, fixed-feature closed-form heads and additional layers cannot" This reads so convoluted, why are we talking about multi-branch in-context relations, what are they? compressive phase retrieval, lifted inner loop? these terms are just thrown at the reader, what reason are they mentioned for?

Next, whether "a PT model can host such a loop" what loop? "We prove six requirements" for what? And then the abstract just keeps adding new and new terms without making any sense. I understand abstract can have some technical terms, but it needs to give some high level idea, but this is so poorly written it's very hard to make sense out of it.

Next the intro second line: "fast weight programmers and test time regression place many sequence models in same design space" what design space? Then says natural step is self-iteration, self-iteration over what?

Second para talks about multi-branch in-context relations, they introduce some labelling function (which is not clear what it is) and then says "a sign-lost function of a task specific direction plus part of attribute the base does not encode; phase retrieval has same form" I'm sorry this in its current form makes no sense, sign lost function what is that, why are we discussing phase retrieval here how are they connected to y

The following discussion on why self-iteration is again just gibberish: "lifted inner loop, each context gets an activity that is inverted globally in one dimension, multipliers coordinate their activities with shared memory, and memory is refit by local least squares" this is quite clearly LLM slop, one can say without needing any tools to check. After this point the reader is completely lost imo bc the problem and motivation just isn't there.

Moving to their setup, no definition of what A and R mean. Text says θ\theta, solver keys and readout are trained on training tasks. Other than θ\theta, we don't know what they are.

Next the text writes what they compare: inner loops with identical keys, data and outer training.....and lifted self-iterating loop" this is written in such a compressed manner, even after multiple passes one can't understand what they're saying. What are they training, why are training, what they are comparing, what is the motivation, why is it relevant, all unclear.

In this form, I have no choice but to recommend rejecting this paper.

Questions for authors

I don't have any questions. One would have to completely re-write the entire paper for it to become readable to humans, and only then one can start asking any questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.