acceptodds
Under review as a conference paper at ICLR 2027

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

29 views

Abstract

Visual backbones have evolved from Convolutional Neural Networks (CNNs) with local aggregation to Vision Transformers (ViTs) with global interactions, State-Space Models (SSMs) with input-dependent state transitions, and Test-Time Training (TTT) layers that adapt an inner learner while processing an image. Across this progression, visual computation has become increasingly adaptive to each input, yet the rules governing that adaptation remain largely prescribed by the trained backbone. We introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. Building on the self-referential construction of Nested Learning (NL), VisionHOPE realizes this co-evolution through five coupled memories that store content, generate key and value representations, and govern learning rate and retention. These memories evolve jointly as visual context accumulates along each scan. However, directly applying the unconstrained self-referential update to a visual backbone leads to instability. We therefore derive a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition, and prove that the resulting memory dynamics are non-expansive along each scan. For two-dimensional feature maps, we adapt NL's chunk formulation by aligning chunks with image rows and columns across four directional scans. VisionHOPE achieves strong results on ImageNet-1K, COCO, and ADE20K, establishing self-modifying learning systems as a practical foundation for general-purpose visual backbones. The code is available at https://anonymous.4open.science/r/ICLR_27_ID_25983.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Representation Learning, Self-modifying Learning, Nested Learning, Visual Backbones

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JC5VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VisionHOPE, a visual backbone based on Nested Learning that enables both stored representations and their learning dynamics to adapt during image processing. The model uses five coupled memories to regulate content, key/value representations, learning rate, and retention. To address instability from self-referential updates, the authors develop a stability-matched step-size mechanism with theoretical non-expansion guarantees. Experiments demonstrate competitive performance on ImageNet-1K, COCO, and ADE20K.

Critical strengths

The strongest contribution is the combination of self-modifying memory dynamics with a mathematically grounded stability mechanism. The experimental results are competitive across several computer vision benchmarks.

  1. The paper explores an interesting direction by extending adaptive visual computation from feature representations to the mechanisms governing their updates.
  2. The proposed stability control is mathematically well motivated, with theoretical guarantees for both token-wise and chunk-wise memory dynamics.
  3. The experimental evaluation is comprehensive, covering multiple visual recognition tasks, backbone architectures, and model scales. The ablation studies and efficiency analyses further support the proposed design.
Critical weaknesses

My main concern is whether the paper convincingly demonstrates the unique advantage of self-modifying learning, beyond the incremental performance improvements obtained by introducing additional adaptive mechanisms into an already strong visual backbone.

  1. Limited evidence for the unique benefit of self-modification. Performance gains over strong baselines are generally modest, and experiments do not directly demonstrate advantages beyond conventional adaptive representations.
  2. Insufficient mechanistic understanding. The functional roles and interactions of the five memories remain unclear.
  3. Unexplored stability–adaptability trade-off. The non-expansion constraints and shared memory updates may limit independent memory evolution and adaptation capacity.
  4. Additional architectural complexity is not fully justified by the relatively small performance improvements.
Questions for authors

I encourage the authors to more directly investigate the functional significance of self-modifying memory dynamics. Experiments involving distribution shifts, sequential visual context, or changing image statistics could help distinguish this mechanism from conventional adaptive feature processing. Quantitative analyses of memory representations and their evolution would also strengthen the mechanistic interpretation. Finally, a more systematic examination of the relationship between stability constraints, memory capacity, and performance would improve the theoretical and empirical contribution.

6 comments
AuthorsOpenReview ↗

[1/6] Response to Reviewer JC5V

Thank you for your positive review. We appreciate your recognition of combining self-modifying memory dynamics with mathematically grounded stability control as the strongest contribution of our work. You also highlight the value of adapting the mechanisms that govern feature updates and the theoretical guarantees for both token-wise and chunk-wise dynamics. We further appreciate your positive assessment of our competitive results and comprehensive evaluation across visual tasks, backbone architectures, and model scales, including the ablations and efficiency analyses that support our design.


W1: Functional contribution of self-modification

Our results show that self-modification improves accuracy, reduces sensitivity to misleading cues, and strengthens long-range perception compared with conventional adaptive representations.

  • Strong results on mature benchmarks. ImageNet-1K, COCO, and ADE20K are mature, highly competitive benchmarks. At the Tiny, Small, and Base scales, VisionHOPE achieves the best or tied-best results among the backbones compared in Tables 1-4. P-VisionHOPE-T improves over ViTTT-T by 1.9 percentage points on ImageNet-1K. VisionHOPE-T improves over H-ViTTT-T by 0.6 box AP and 0.3 mask AP on COCO and 1.4 mIoU on ADE20K.

  • Direct evidence on the benefits of self-modification. Our P-VisionHOPE-S ablations in Table 5 keep the backbone and training protocol fixed, and the content memory continues to adapt in every variant. We compare the full Self-Referential Nested Learning (SRNL) model with variants that fix key/value (K/V) mappings, update dynamics, or both within each image. Adapting the learning mechanism adds 0.6 percentage points over adapting content alone.

    Variantw/o adaptive K/V & dynamicsw/o adaptive K/Vw/o adaptive dynamicsFull model
    Top-1 Acc (%)81.781.982.082.3
  • Controlled comparison with other adaptive representations. ViTTT updates an inner learner under a fixed learning rule, providing adaptive visual representations without self-modification. We replaced the VisionHOPE operator in P-VisionHOPE-T/S/B with the original ViTTT operator, including its full-image inner adaptation. All other P-VisionHOPE backbone components and the training protocol were kept unchanged. The table reports full-model costs and ImageNet-1K Top-1 accuracy at 224×224. With comparable parameter counts and FLOPs, VisionHOPE improves Top-1 accuracy by 1.1, 0.9, and 0.7 percentage points at the Tiny, Small, and Base scales, respectively.

    ScaleOperatorParams (M)FLOPs (G)Top-1 Acc (%)
    TinyViTTT61.277.3
    TinyVisionHOPE61.278.4
    SmallViTTT234.681.4
    SmallVisionHOPE224.782.3
    BaseViTTT8918.882.7
    BaseVisionHOPE8819.083.4
  • Smaller performance drop under misleading context. We compared P-VisionHOPE-S with Vim-S, which also uses adaptive visual representations. Each scan began with memory inputs from an image of another class, followed by inputs from the original image. On the full ImageNet-1K validation set, the Top-1 drop was 0.0 percentage points for P-VisionHOPE-S and 0.2 points for Vim-S.

  • Stronger long-range perception. Figure 5 compares effective receptive fields (ERFs) across visual backbones. DeiT-S and ViTTT-S concentrate more strongly around the center, while Vim-S shows a pronounced directional pattern. P-VisionHOPE-S combines a clear central response with broader coverage across the image, showing stronger long-range perception.

Vim keeps the mappings that generate its state transitions fixed within each image, while ViTTT updates an inner learner under a fixed learning rule. VisionHOPE updates the mappings that generate learning targets and govern memory updates, allowing visual context to change the learning rule itself. The higher accuracy, smaller drop under misleading cues, and stronger long-range perception demonstrate the practical advantages of this self-modification.

AuthorsOpenReview ↗

[2/6] Response to Reviewer JC5V

W2: Roles and interactions of the five memories

The five memories have distinct roles and are coupled through the learning signals they generate.

MemoryFunction in the coupled system
ContentMaps queries to memory outputs
KeyGenerates keys that determine writing addresses and correction directions
ValueProduces the shared representation each memory uses to form its learning target
Learning rateGenerates the raw step size before stability control
RetentionControls how much of the previous state is retained and limits self-referential feedback

The key and value memories supply the writing addresses and shared representation for all five updates. Each memory applies its own current mapping to that representation to form its learning target. The learning-rate and retention memories regulate these updates under the stability controls. All five memories are updated, and their new states generate the signals for later learning. Earlier visual context can therefore change both the stored content and the learning rule applied to subsequent inputs.


W3: Stability control and shared updates

Stability control enables stable training while allowing the memories to continue evolving. Shared updates preserve the coupled evolution of the five memories.

1. Memory evolution under stability control

  • Stability control is necessary for training. The stability ablations in Table 5 show that uncontrolled self-referential feedback causes early numerical divergence (NaN). Stability control is therefore a prerequisite for trainable self-modification. With stability control, P-VisionHOPE-S trains successfully and reaches 82.3% Top-1 accuracy. We therefore cannot directly compare memory evolution in trained models with and without stability control. In Q3 below, we compare different stability-control schemes and show that our combination achieves the highest Top-1 accuracy among the tested schemes.

  • Memory evolution continues under stability control. Figure 6 shows the executed step sizes η~t\widetilde{\eta}_t used to update the five memories in the last P-VisionHOPE-S block. These steps vary across image locations, with larger values often aligned with prominent objects and distinctive visual structures. The memories therefore continue to evolve through controlled updates that respond to visual content.

2. Memory evolution under shared updates

The five memories already evolve as a coupled system under the linear self-referential update adopted from Nested Learning (NL). Each memory has its own functional mapping and forms its own learning target. Equation (14) and Proposition 2 express this update using an equivalent shared factor. The key, value, learning-rate, and retention memories generate the quantities that determine this factor. Applying it updates each memory's own mapping, including those that generate future learning signals. The updated mappings then determine the learning signals and shared factor for later inputs. Sharing the factor therefore preserves both the evolution of the individual memory mappings and the feedback between them.

AuthorsOpenReview ↗

[3/6] Response to Reviewer JC5V

W4: Architectural complexity and measured benefit

VisionHOPE achieves substantial gains on mature visual benchmarks without increasing the architectural complexity of the surrounding backbone. It uses its own core operator within a standard residual architecture, with parameter counts and FLOPs comparable to those of other baselines.

  • A core operator in a standard residual architecture. VisionHOPE provides the core computation in each block, just as attention, state-space operations, or TTT operators do in other backbones. It is not an auxiliary module stacked on another backbone. The surrounding architecture uses standard projections, normalization, residual connections, and a feed-forward network (Section 2.3 and Figure 2). The operator preserves the feature-map shape. We have made our implementation available, with modular components and a simple operator interface.

  • Comparable parameter counts and FLOPs. Tables 1-4 report full-model costs at the Tiny, Small, and Base scales. VisionHOPE's parameter counts and FLOPs are comparable to those of other baselines at the corresponding scales. For example, P-VisionHOPE-S reaches 82.3% Top-1 with 22M parameters and 4.7G FLOPs, compared with 81.8% at 26M and 5.3G for Vision-TTT-S.

  • Measured throughput and memory efficiency. Figure 3 shows that P-VisionHOPE-B achieves higher throughput and lower memory use than Vim-B across all tested resolutions from 224×224224 \times 224 to 1280×12801280 \times 1280. Figures 3 and 4 also show higher throughput and lower memory use than DeiT at high resolutions across the Tiny, Small, and Base scales.

  • Substantial gains on mature benchmarks. ImageNet-1K, COCO, and ADE20K are mature, highly competitive benchmarks. Our backbones achieve the best or tied-best results at the Tiny, Small, and Base scales among the models compared in Tables 1-4. The performance gains are detailed in W1. The controlled operator comparison keeps the surrounding backbone and training protocol unchanged. With comparable parameter counts and FLOPs, VisionHOPE improves ImageNet-1K Top-1 over ViTTT by 1.1, 0.9, and 0.7 percentage points at the Tiny, Small, and Base scales, respectively.

  • Ablations justify the core design choices. Table 5 evaluates the adaptive memories, Delta Gradient Descent (DGD) correction, stability control, scan directions, and fusion. The full model achieves 82.3% Top-1. Removing the DGD correction gives 81.9%, using one scan direction gives 81.6%, and replacing learned fusion with a fixed sum gives 82.1%. The memory and stability ablations also show the benefits of adapting the learning mechanism and the need to control its feedback. These ablations show that the core design choices improve accuracy and enable stable self-modifying learning.

Reviewer_eAK9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In the current paper, the authors introduce VisionHOPE, the first generic visual backbone formulated as a self-modifying learning system, in which what the model remembers and how it learns co-evolve within an image. This co-evolution is achieved through five coupled memories (content, key, value, learning rate, and retention) operating across four directional scans with spatially aligned chunks to achieve linear scaling with the token count. To prevent the numerical instability typically caused by unconstrained self-referential updates, the framework implements a stability-matched step-size control scheme combining a soft injection cap and a spectral clamp. This design guarantees non-expansive memory dynamics across both token-wise and chunk-wise recurrences. This design guarantees non-expansive memory dynamics across both token-wise and chunk-wise recurrences. Comprehensive evaluations demonstrate that VisionHOPE achieves strong performance on ImageNet-1K classification, COCO object detection and instance segmentation, and ADE20K semantic segmentation tasks.

Critical strengths

Here are the paper's critical strengths:

  • VisionHOPE introduced the first generic visual backbone formulated as a self-modifying learning system, enabling stored content and the learning rule to co-evolve within an image
  • it couples five different types of memories (content, key, value, learning rate, and retention) allowing comprehensive adaptation of both what is remembered and how it is learned
  • the authors mathematically prove and guarantee non-expansive memory dynamics along both token-wise and chunk-wise recurrences
  • unlike standard vision transformers that materialize quadratic attention matrices, VisionHOPE maintains linear activation-memory scaling as input resolutions increase
Critical weaknesses

Here are the paper's critical weaknesses:

  • directly applying unconstrained self-referential updates without stability controls leads to early numerical divergence and NaN errors during training.
  • the framework's training stability heavily depends on the specific combination of the soft injection cap and spectral clamp
  • the model requires careful tuning of multiple internal parameters, such as the raw-step scale (γη\gamma_{\eta}), initial retention (αinit\alpha_{init}), head dimensions (dd), and internal width ratios (Dm/DD_m/D).
  • running independent SRNL instances across four separate scan directions introduces structural complexity compared to single-direction models
  • while the theoretical formulation of VisionHOPE is elegant and mathematically rigorous, the empirical improvements over existing state-of-the-art baselines are relatively modest given the substantial structural complexity introduced
Questions for authors

These are my concerns:

  • VisionHOPE couples five different types of memories (content, key, value, learning-rate, and retention), raising questions about their individual contributions. Can the authors provide ablations assessing whether all five are necessary?
  • the paper mentions that the spectral clamp is triggered in only 0.0044%0.0044\% of token-wise updates during inference. Given that this intervention is exceptionally rare, does it actively prevent instability during normal training? Or could it be removed without affecting the model's convergence and accuracy?
  • the paper sets the initial retention to αinit=0.9\alpha_{init} = 0.9 and the raw-step scale to γη=0.025\gamma_\eta = 0.025 without detailing how they were chosen. Can the authors provide a sensitivity analysis showing how model performance and training stability respond when these two parameters are varied across a reasonable range?
  • table 10 indicates that hierarchical VisionHOPE models use MESA training to boost accuracy. Since many baseline models are trained without MESA, what is the exact performance of VisionHOPE-T/S/B without MESA (on ImageNet-1K) to ensure a fair comparison?
6 comments
AuthorsOpenReview ↗

[1/4] Response to Reviewer eAK9

Thank you for your thoughtful review and positive assessment of our work. We appreciate your recognition of VisionHOPE as the first generic self-modifying visual backbone, whose five coupled memories enable stored content and the learning rule to evolve within each image. You also highlight our non-expansion guarantees for token-wise and chunk-wise recurrences and linear scaling of activation memory with image resolution.


W1: Numerical stability of the coupled update

  • Theoretical origin and visual implementation. VisionHOPE adopts the linear self-referential update from the Nested Learning (NL) framework of Behrouz et al. (2025). At the time of our work, no official NL implementation was publicly available. We implemented this formulation for visual backbones and observed numerical divergence when self-referential feedback was left uncontrolled.

  • Source of numerical instability. Our analysis in Section 2.2 identifies the source of this instability. The adopted self-referential update does not constrain the memory transition to be non-expansive. Equation (15) shows that the transition can amplify the memory state. The updated memories then generate the quantities governing subsequent updates, allowing this amplification to feed back and compound along the scan.

  • Our contribution to stable training. We therefore derive complementary bounds on the retained transition and self-referential injection, and enforce them through a soft injection cap and a spectral clamp. These controls guarantee non-expansive memory dynamics for both token-wise and chunk-wise recurrences. Table 5 demonstrates their practical effect. Removing the soft injection cap causes early numerical divergence (NaN), while the full model trains successfully and reaches 82.3% Top-1 accuracy. Deriving these stability conditions and enabling stable training of a self-modifying visual backbone are central contributions of VisionHOPE.


W2: Roles of the two stability controls

The two controls limit amplification in different parts of the memory update.

  • Soft injection cap. Self-referential feedback can amplify the evolving memory states. The soft cap bounds this feedback by the contraction margin left by retention. The capped step remains responsive to the raw step generated by the learning-rate memory, even near the injection limit.

  • Spectral clamp. The retained transition can also become expansive when the step is too large. The spectral clamp enforces the corresponding step-size limit. It acts only on candidate steps that exceed this limit and leaves all other candidates unchanged. A soft spectral cap instead remaps every positive candidate step.

Table 5 evaluates alternative control combinations using the same P-VisionHOPE-S backbone and training protocol:

Step-size controlTop-1 (%) / training outcome
Without soft injection capNaN
Without spectral clamp82.3
Injection clamp + spectral clamp81.8
Soft injection cap + soft spectral cap81.9
Soft injection cap + spectral clamp82.3

Table 5 shows that P-VisionHOPE-S trains successfully with several control combinations. Replacing the soft injection cap with an injection clamp or the spectral clamp with a soft spectral cap lowers accuracy to 81.8% and 81.9%, respectively. Removing the spectral clamp retains 82.3% accuracy, consistent with its rare activation in Appendix C.5. The non-expansion proof requires the spectral norm of the retained transition to be at most αt\alpha_t and that of the self-referential injection to be below 1−αt1-\alpha_t. The soft injection cap alone does not guarantee the retained-transition bound. We therefore retain the spectral clamp to enforce this remaining condition at every update, completing the theoretical non-expansion guarantee.

AuthorsOpenReview ↗

[2/4] Response to Reviewer eAK9

W3: Configuration sensitivity

  • Shared defaults across six backbones. All hierarchical and plain VisionHOPE models at the Tiny, Small, and Base scales use the same Self-Referential Nested Learning (SRNL) settings: raw-step scale γη=0.025\gamma_\eta=0.025, initial retention αinit=0.9\alpha_{\mathrm{init}}=0.9, head dimension d=16d=16, and internal-width ratio Dm/D=1/2D_m/D=1/2.

  • Capacity and computational cost. Dm/DD_m/D controls the internal feature width, and dd sets the dimension of each memory head. These architecture parameters can be adjusted to the desired model capacity and computational budget.

  • Low sensitivity of the update parameters. Table 9 varies γη\gamma_\eta from 0.0125 to 0.1, an eightfold range, while Top-1 accuracy remains between 82.1% and 82.3%. The additional results for αinit\alpha_{\mathrm{init}} discussed in Q3 likewise show only small accuracy changes across the tested settings.

These results suggest that the reported VisionHOPE backbones can use a common set of default update parameters without extensive tuning for each model.


W4: Four-direction scanning for visual context

  • Why multiple scan directions matter. In a single causal scan, a token can receive recurrent context only from tokens earlier in the scan order. Tokens later in that order cannot contribute through the same scan. Vim (Zhu et al., 2024) addresses this limitation with bidirectional state-space modeling. VMamba (Liu et al., 2024) further reports that scanning in four directions outperforms bidirectional scanning. Its four routes traverse rows and columns in both directions, providing complementary context aligned with the two-dimensional image structure.

  • Evidence from our directional ablations. Table 5 compares one, two, and four scan directions in P-VisionHOPE-S under the same backbone and training protocol. Their Top-1 accuracies are 81.6%, 81.9%, and 82.3%, respectively. Four directions therefore improve accuracy by 0.7 percentage points over one direction and 0.4 points over two directions.

The complementary spatial context and accuracy gains in Table 5 justify the additional structure introduced by four-direction scanning.


W5: Accuracy gains and practical efficiency

ImageNet-1K, COCO, and ADE20K are mature, highly competitive benchmarks. At the Tiny, Small, and Base scales, our backbones achieve the best or tied-best results among the models compared in Tables 1-4.

  • Gains over SOTA visual backbones. On ImageNet-1K, P-VisionHOPE improves over Vision-TTT by 0.7, 0.5, and 0.7 percentage points at the Tiny, Small, and Base scales, respectively (Table 2). P-VisionHOPE-T also improves over ViTTT-T from 76.5% to 78.4%, a 1.9-point gain. On ADE20K, VisionHOPE-T improves over H-ViTTT-T from 48.0 to 49.4 mIoU, a 1.4-point gain (Table 4). On COCO, VisionHOPE-T improves over H-ViTTT-T from 47.3 to 47.9 box AP and 42.8 to 43.1 mask AP (Table 3).

  • Competitive parameter counts and FLOPs. The comparisons use corresponding model scales. For example, P-VisionHOPE-S achieves 82.3% Top-1 with 22M parameters and 4.7G FLOPs, compared with 81.8% at 26M and 5.3G for Vision-TTT-S. On ADE20K, the complete UPerNet model with VisionHOPE-T uses 55M parameters and 942G FLOPs, compared with 58M and 946G with H-ViTTT-T. In both comparisons, VisionHOPE achieves higher accuracy with fewer parameters and lower FLOPs.

  • Measured throughput and memory use. Figure 3 shows that P-VisionHOPE-B achieves higher throughput and lower memory use than Vim-B across all tested resolutions from 224×224224 \times 224 to 1280×12801280 \times 1280. Figures 3 and 4 also show higher throughput and lower memory use than DeiT at high resolutions across the Tiny, Small, and Base scales.

  • A new self-modifying visual architecture. VisionHOPE establishes a stable and efficient self-modifying visual backbone based on NL. We have made our implementation available, with modular components that are easy to use and extend.

AuthorsOpenReview ↗

[3/4] Response to Reviewer eAK9

Q1: Individual contributions of the five memories

1. Grouped ablations in Table 5

Table 5 already evaluates the grouped contributions of the key/value memories and the learning-rate/retention memories (dynamics) in P-VisionHOPE-S. Disabling adaptive K/V reduces Top-1 accuracy from 82.3% to 81.9%, disabling adaptive dynamics gives 82.0%, and disabling both gives 81.7%.

2. Individual memory ablations

We additionally ablate each memory separately in P-VisionHOPE-S. Each variant fixes one memory at its learned initial parameters throughout the image, while the other four memories continue to update.

VariantMemory roleTop-1 Acc (%)
w/o adaptive contentStores content and produces memory outputs80.6
w/o adaptive keyGenerates keys for memory updates82.1
w/o adaptive valueGenerates representations used to form each memory's target82.0
w/o adaptive learning rateGenerates raw update step sizes82.1
w/o adaptive retentionControls retention and the available injection margin81.9
Full modelJoint evolution of all five memories82.3

Disabling the online adaptation of any individual memory lowers accuracy. Content adaptation contributes the largest gain of 1.7 percentage points, while the gains from adapting the other four memories range from 0.2 to 0.4 points. These results support the importance of online adaptation in all five memories.


Q2: Spectral-clamp activity during training and its removal

1. Training observations and removal ablation

We observed no NaNs when training our visual models without the spectral clamp. On ImageNet-1K, we obtained 82.3% Top-1 accuracy with and without the clamp using P-VisionHOPE-S (Table 5). Thus, this configuration trains successfully without it and retains the same accuracy. We also found that the clamp activates in only 0.0011% of token-wise updates over the first ten warmup epochs and 0.0044% during inference (Appendix C.5), consistent with its limited effect in our experiments.

2. Completing the non-expansion guarantee

We retain the spectral clamp to complete the theoretical non-expansion guarantee. The soft injection cap bounds self-referential feedback but does not by itself enforce the retained-transition condition η~t∥kt∥22≤2αt\widetilde{\eta}_t\lVert k_t\rVert_2^2\leq 2\alpha_t. Without this condition, repeated expansive updates can accumulate into numerical divergence. The spectral clamp enforces this condition and leaves candidate steps below the limit unchanged.

3. A constructed example of numerical divergence

We tested the chunk-wise five-memory recurrence with constructed initial states and input sequences. The inputs were designed to repeatedly produce near-zero retention with candidate steps above the spectral limit. Both runs used identical initial states and inputs with the soft injection cap enabled. They differed only in whether the spectral clamp was applied.

Spectral clampWithoutWith
OutcomeNaN at token 28,757Finite throughout

Without the spectral clamp, repeated amplification led to Inf and then NaN. With it, all five memory norms stayed at or below their initial values throughout the sequence.

Reviewer_zEeCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VisionHOPE, a visual backbone formulated as a self-modifying learning system. what it remembers and how it learns co-evolve within an image. Besides, the paper also introduces a stability-matched step-size control scheme that combines a soft cap on self-referential injection with a spectral clamp on the retained memory transition. VisionHOPE achieves good results on ImageNet-1K, COCO, and ADE2OK.

Critical strengths

The experiments cover image classification, object detection and instance segmentation, and semantic segmentation on ImageNet-1K, COCO, and ADE20K.

Critical weaknesses
  1. The writing needs improvements. For example, the motivation for extending TTT with five coupled memories is not sufficiently clear. The introduction doesn't explain what limitations TTT have and which is solved.
  2. The paper overstates its claims. It emphasizes that the five memories are coupled and co-evolve so the model can modify its own learning rule, but as shown in Eq. (14) and Proposition 2, all five memories share the same right-multiplicative update. This means that they actually use the same dynamic transformation for updates and each memory only needs to store its own initial parameters. And the results in Table 5 also show this (w/o adaptive K/V & dynamics and w/o adaptive dynamics only get very limited improvement).
  3. Theorem 1 only proves that the memory norm is non-increasing, which does not characterize how quickly information is forgotten along the scan. With αinit=0.9\alpha_{\text{init}} = 0.9, orthogonal directions retain only about 90% of their magnitude per step, leaving just about 0.3% after a single row of 56 tokens. Besides, the injection limit (1−αt)/rt(1-\alpha_t)/r_t ties stronger retention to weaker self-referential updates. It remains unclear how much long-range information each scan actually carries, and how much self-modification survives once the stability control is applied.
  4. The ablation study is insufficient to tell whether the gains actually come from the self-modifying learning itself or from other components like convolutions.
Questions for authors
  1. What's the advantages compared with the TTT update method? Can authors compare against a TTT-style update under the same block and training protocol to show that this additional adaptivity is actually necessary?
  2. How well does it recover when an early visual clue is misleading?
9 comments
AuthorsOpenReview ↗

[1/5] Response to Reviewer zEeC

Thank you for your insightful review. We appreciate your recognition of the breadth of our experimental evaluation, covering image classification, object detection, instance segmentation, and semantic segmentation on ImageNet-1K, COCO, and ADE20K.

Our goal is to open another direction for visual backbone research alongside ViTs, SSMs, and TTT models. We build on the Nested Learning (NL) framework of Behrouz et al. (2025) to introduce the first generic visual backbone formulated as a self-modifying learning system. Both its stored content and the mechanisms that update it evolve within each image. We offer this foundation for the community to explore and develop, without claiming that one architecture should outperform every existing approach in every setting.

At the time of our work, no official NL implementation was publicly available. We implemented the self-referential learning system and derived stability controls with non-expansion guarantees. We then designed an efficient visual operator and built both hierarchical and plain backbones around it. We evaluated these models across scales and standard visual recognition tasks, and made our implementation available. This work provides the theory, code, and experimental evidence for further research on self-modifying visual backbones.

Below, we explain why sharing the transition preserves self-modification, quantify memory retention and self-referential injection under stability control, and show the gains from adapting the learning mechanism.


W1: Motivation for self-modifying learning in vision

VisionHOPE is derived from the self-referential formulation of NL, not from extending a TTT layer to five memories. NL views a model as a set of connected learning processes. As described in the Introduction and Section 2.1, we use this principle to let a visual backbone learn both what it stores and how it updates its memory.

  • Limitation of TTT and what NL achieves. The visual TTT models discussed in Appendix A.3 update an inner learner as they process an image. The mappings that produce its keys and values and control its updates remain largely fixed. NL allows these mappings to learn as well. Earlier context can then change how the model learns from later inputs. VisionHOPE implements this form of adaptation within each image.

  • How the five memories interact. Replicating TTT across five memories does not define how they learn together. Our Self-Referential Nested Learning (SRNL) module follows NL to specify these interactions. The key memory provides writing addresses, and the value memory produces a shared representation. Each memory uses its own current mapping to turn that representation into its learning target. The learning-rate and retention memories control all five updates. The updated states then generate the learning signals for later updates.

  • Contribution to visual backbone design. This feedback system needs stable updates and efficient computation on two-dimensional feature maps. Our stability-matched step-size controls guarantee non-expansion for both token-wise and chunk-wise updates. Four directional scans with spatially aligned chunks organize learning over image rows and columns. The operator supports both hierarchical and plain backbones and scales linearly with the number of visual tokens.

We will revise the Introduction and Related Work to clarify these limitations of visual TTT models. The revised text will also make clear that VisionHOPE is a self-modifying visual backbone based on NL.

AuthorsOpenReview ↗

[2/5] Response to Reviewer zEeC

W2: Self-modification and its measured benefits

We have not overstated the self-modifying capabilities of VisionHOPE.

1. The shared factor carries the evolving memory state

VisionHOPE adopts the linear self-referential update from NL. Substituting the self-generated learning signals into Eq. (10) gives the equivalent form in Eq. (14), and Proposition 2 establishes the corresponding shared-factor representation under chunking. The shared right factor is generated by the evolving memories, and applying it changes the memories that generate subsequent updates.

For a chunk spanning tokens b+1,…,b+Cb+1,\ldots,b+C, Eq. (12) generates the update quantities from the boundary memories. Including the key normalization specified in Appendix C.1, these quantities are

k_t=\frac{k_t^{\mathrm{raw}}}{\sqrt{\lVert k_t^{\mathrm{raw}}\rVert_2^2+10^{-6}}},\qquad v_t=M_b^v x_t,\qquad \eta_t=\phi_\eta(m_b^\eta x_t),\qquad \alpha_t=\phi_\alpha(m_b^\alpha x_t).$$ Each memory $\square\in\lbrace{m,k,v,\eta,\alpha\rbrace}$ forms its own target and learning signal as in Eq. (13). With $\delta_t=k_t-v_t$, this gives $$\widehat v_t^\square = M_b^\square v_t, \qquad g_t^\square = M_b^\square k_t-\widehat v_t^\square =M_b^\square\delta_t.$$ Applying the stability controls in Eqs. (16) and (17) gives the executed step $\widetilde\eta_t$. Substituting the learning signal into Eq. (10), with the operators defined in Eq. (18), yields the recurrence in Appendix B.4: $$A_t = \alpha_t I_d-\widetilde\eta_t k_tk_t^\top, \qquad B_t = -\widetilde\eta_t\delta_tk_t^\top, \qquad M_t^\square = M_{t-1}^\square A_t+M_b^\square B_t.$$ Thus, the key, value, learning-rate, and retention memories jointly determine the update operators, while each memory's own boundary state determines its learning target. Proposition 2 factors this recurrence as $$M_t^\square = M_b^\square G_t, \qquad G_b = I_d,\qquad G_t=G_{t-1}A_t+B_t.$$ **The feedback becomes explicit at the next chunk.** Substituting $M_{b+C}^\square=M_b^\square G_{b+C}$ into Eq. (12), for a token $u\in\lbrace{b+C+1,\ldots,b+2C\rbrace}$, gives $$k_u^{\mathrm{raw}} = M_b^k G_{b+C}x_u,\qquad v_u=M_b^v G_{b+C}x_u, \qquad \eta_u = \phi_\eta(m_b^\eta G_{b+C}x_u),\qquad \alpha_u=\phi_\alpha(m_b^\alpha G_{b+C}x_u).$$ The key $k_u$ is obtained by the same normalization above. These quantities determine the next chunk's $A_u$ and $B_u$, and hence its shared factor. The preceding factor therefore changes both the content memory and the mappings that generate subsequent learning signals. The updated memories also form their next targets through Eq. (13), completing the feedback across chunks. **Representing the memories by their initial parameters and an accumulated right factor preserves this entire dependence.** The initial parameters specify the starting states, while the accumulated factor carries the evolving state needed to generate subsequent learning signals and updates. ### 2. Coupled memory evolution and self-modification - **The memories are coupled through their learning signals.** Each memory is updated using quantities generated by other memories in the system. The key and value memories provide writing addresses and target representations, while the learning-rate and retention memories control all five updates. Each memory forms its own target by applying its mapping to the value representation. **The update of the content memory therefore depends on the current states of the other four memories.** - **The shared transition implements co-evolution.** The five memory maps change together as the image is processed. The common transition coordinates these changes while preserving their different functional roles. In Eq. (14), each new transition depends on the states reached through earlier updates. - **The model modifies its own learning rule.** The maps that generate learning targets and update dynamics are themselves updated during image processing. Their changed parameters determine how later observations update the content memory. **VisionHOPE therefore learns both its stored content and the mappings that control subsequent learning.** This is the self-modification described by NL and implemented in our visual backbone.
AuthorsOpenReview ↗

[3/5] Response to Reviewer zEeC

W2: Self-modification and its measured benefits

3. Accuracy gains from adapting the learning mechanism

Table 5 evaluates the two ablations highlighted in the review using the same P-VisionHOPE-S backbone and training protocol. In these controls, the specified key/value or dynamics mappings remain fixed within the image, while the content memory continues to adapt.

Variant in Table 5w/o adaptive K/V & dynamicsw/o adaptive dynamicsFull model
ImageNet-1K Top-1 (%)81.782.082.3

Full SRNL gains 0.6 percentage points over freezing K/V and dynamics, and 0.3 points over freezing dynamics alone. Both gains come on top of an already adaptive content memory and measure the benefit of learning the mappings that govern its updates. ImageNet-1K is a mature benchmark where strong backbones at a comparable scale can differ by only a few tenths of a percentage point. These are substantial gains from adapting the learning mechanism within the same architecture and training recipe.

We will add a Discussion section to explain how the shared updates implement coupled self-modification and what the controlled ablations show about the value of adapting the learning mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.