acceptodds
Withdrawn from ICLR 2027

CCPL: Continuous Causal Predictive Learning for Spatially Queryable Visual Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce Continuous Causal Predictive Learning (CCPL), a self-supervised framework that accumulates a causal state from local glimpses and predicts a continuous-valued next-glimpse latent conditioned on real-valued relative pan and zoom through a conditional flow matching objective. Unlike masked or autoregressive image models, whose predictions are specified by a mask position on a fixed patch grid or a fixed scan order, CCPL conditions each prediction on a continuous relative move, and hence the frozen model can be queried at independently specified positions and scales. After pretraining on ImageNet, query error decreases as a prescribed random-walk history grows and increases when recurrent memory is reset or query-move conditioning is shuffled. These controls provide operational evidence that the causal state accumulates information useful for spatial prediction across the sequence. We then freeze the observer and predictor and train an explorer with a category-label-free terminal reward given by normalized query error from the final feature. At a fixed budget of 1024 glimpses, learned scans reach the lowest terminal query error among all evaluated scans on both CCPL models and raise final-feature linear-probe top-1 over matched random walks by 1.18 and 1.34 points. However, recognition-leading lattice scans have higher query MSE than random walks. On the learned-scan's final and mean features, two-layer probes raise top-1 by 9 to 11 points over the corresponding linear probes, suggesting that class information in CCPL features is better accessed nonlinearly. Thus, a frozen predictive model can supply a category-label-free acquisition signal, but reconstruction and recognition favor different acquisition strategies and require separate evaluation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self Supervised Learning, Computer Vision, Vision Encoder

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6KemOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

A self-supervised learning framework that accumulates causal memory through continuous local observation and predicts features at any position and scale. Experiments show that a longer observation history can reduce prediction errors, and the label-free exploration strategy based on prediction errors can also effectively optimize information acquisition. However, the strategy with the minimum prediction error does not always have the best classification performance, which indicates that the prediction and recognition tasks require different search strategies.

Critical strengths

1.The overall idea of the paper is relatively easy to understand. The model views the image block by block and predicts the scene it will see based on what it has seen and the next move. Figure 1 compares it side by side with existing methods, clearly illustrating the differences between it and the existing methods.

2.The overall content of the paper is rich, and the appendix also provides a large number of supplementary experiments and details. These supplements provide more comprehensive support for the conclusion of the paper.

Critical weaknesses

1.This experiment does not reflect the value of the core concept. The pre-training of random walk on 256x256 images leads to a large number of repeated visits to previously seen areas (with coverage increasing from 21% in 64 steps to 98% in 1024 steps). “Predicting the next patch” is not about guessing unknown areas, but simply recalling. It is not sufficient to prove the spatial reasoning ability claimed by the paper.

2.The performance is extremely low. The linear probing accuracy of CCPL on ImageNet is only about 20%, and that of a two-layer MLP is less than 40%, lagging far behind mainstream self-supervised methods such as MAE. Although there are differences in the evaluation protocols, such huge differences make us doubt the actual value of the learned features.

3.The purpose of random walk scanning is unclear. This paper does not discuss its advantages over fixed sequences (raster, spiral, etc.) or scanning methods with fewer repeated visits. Have the model and the explorer learned a reasonable and interpretable sequence of observations?

Questions for authors

Can it be distinguished whether the model's prediction is inferring unknown areas or recalling previously seen content, and can its recognition performance be compared with methods such as MAE under the same protocol? Furthermore, what are the purposes and advantages of random walk scanning, and has the model learned a reasonable viewing sequence?

Reviewer_NuSZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose the CCPL paradigm, which predicts continuous relative displacement targets from causal sequences of local glimpses, thereby departing from conventional rigid scanning regimes. Furthermore, empirical findings reveal a fundamental divergence between visual reconstruction optimization and downstream classification regarding sampling trajectory preferences, demonstrating that an observation sequence yielding superior reconstruction is not necessarily semantically optimal.

Critical strengths

1.The author's finding that observation sequences with good reconstruction are not semantically optimal is counterintuitive, which is quite inspiring to me.

Critical weaknesses

1.Predicting continuous relative displacement targets from causal local glimpse sequences is essentially an adaptive scanning mechanism. However, many adaptive scanning strategies have already been proposed, yet the authors do not provide comparisons with existing adaptive scanning methods. In my view, the overall methodology still fundamentally lacks sufficient novelty.

2.Overall, the figures throughout the manuscript require thorough refinement. Upon my initial reading, I could not glean much informative insight from them: Figure 1 lacks an intuitive motivation, making it difficult to clearly grasp the authors' intended message, while Figure 2 suffers from an cluttered layout where structural flows and mathematical formulations are chaotically intermingled.

3.Regarding the experiments, there are two primary deficiencies. First, it is imperative for the authors to provide a quantitative comparison table of the per-image inference overhead (such as latency, throughput, and computational cost), which is critically missing. Second, the performance gain achieved by RL-based active exploration is excessively marginal, making it difficult to substantiate this component as a core contribution of the proposed algorithm

Questions for authors

See Weaknesses

Reviewer_AyULOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a recurrent visual model to predict a patch at a queried position and scale from a history of glimpses. It then freezes this model and uses its prediction error to train an explorer. Experiments measure spatial-query reconstruction and frozen-feature recognition under learned, random, and fixed scans. The learned scan gives the lowest reconstruction error, but several fixed scans give better recognition accuracy.

Critical strengths
  1. Reusing the frozen predictor to train an explorer is an interesting part of the work. The observer does not need to be retrained for acquisition, and the reward does not use category labels. The policy chooses movements from its current state without access to the target patches. This gives a concrete way to use the pretrained model beyond a recognition probe.

  2. The reconstruction improvement is consistent across the two evaluated observers. It also beats all nine fixed scans on reconstruction (Table 3).

  3. Table 2 checks whether the predictor actually uses history and query location. Resetting the memory raises pixel error from 0.0273 to 0.0830, and shuffling query movements within scale raises it to 0.0931. Longer histories also improve the fully unobserved subset.

Critical weaknesses

I am not convinced that the experiments establish the benefit of the proposed pretraining objective. The observer and predictor are always trained with CCPL; the main comparison changes how that frozen model is scanned. It establishes that acquisition can improve a CCPL model, but not what is gained from CCPL over another way of learning a queryable representation. The paper acknowledges this in Section 4.1, but it remains a substantial gap for a paper proposing a learning formulation.

The learned explorer could be finding a better general movement pattern rather than making useful decisions based on the image. It controls only one in every eight moves, with the remaining moves following the pretraining random walk. The fixed scans differ in both motion and coverage, so they do not isolate adaptation to image content.

The practical gain is less clear than the reconstruction result. On the larger model, final-feature linear accuracy improves from 18.32% to 19.66%, while Spiral reaches 22.60%. Mean-feature accuracy changes from 26.34% to 25.98%, with the reported interval including zero for that decrease. The two-layer head retains the final-feature gain over random scanning but does not remove the fixed scans' lead.

Questions for authors

Can you compare CCPL with an alternative pretraining objective using the same observer, codec, and observation trajectories? This would help establish what the proposed objective adds beyond the acquisition strategy.

Does the learned explorer still improve reconstruction when its trajectories are replayed on different images, or when it is trained without image features? This would clarify whether the gain comes from adapting to image content or learning a better general scan pattern.

Given that fixed scans achieve higher recognition accuracy, when would you recommend using the learned explorer? Does its advantage become clearer at lower observation budgets, where selecting informative glimpses may matter more?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.