acceptodds
Under review as a conference paper at ICLR 2027

Training Convolutional Networks at Temporarily Reduced Internal Resolution

Abstract

Continuation methods train neural networks on a sequence of modified problems before recovering the original objective. We investigate spatial continuation for image classification with convolutional networks: the internal spatial resolution of feature maps is temporarily reduced, then progressively restored during training. This simple intervention improves mean test accuracy in all twenty no-augmentation settings, by 0.38 to 6.28 points, and most of this gain comes from the temporary reduction itself rather than from the progression. The settings span datasets, dataset sizes, architectures, activations, normalizations, optimizers and training budgets. The intervention is absent from the final network, so inference cost is unchanged. Its measured training time is comparable to standard training in the short reference setting and lower in a controlled long-SGD timing study. In matched comparisons with a prior smoothing curriculum, resolution reduction has higher mean accuracy in the no-augmentation recipes, while the augmented SGD comparison depends on BatchNorm calibration. The gains are more consistent with simple optimizers than with adaptive optimizers, where they remain present but vary more. Their magnitude, however, depends on the training recipe, and stronger data augmentation can substantially reduce it. Comparisons with Gaussian filtering of the feature maps, used here as a control, indicate that the gains cannot be attributed to smoothing alone. Geometric diagnostics show lower curvature in the reference setting, but comparisons across methods and training recipes do not support lower curvature as a general explanation of the gains. Overall, these results support temporary internal resolution reduction as a simple and useful intervention in unaugmented training, while showing that its advantage narrows under stronger recipes.

Primary area:
optimization
Keywords:
graduated optimization, continuation methods, curriculum learning, loss landscape, training dynamics, sharpness, convolutional neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xzV9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates spatial continuation for training CNNs (primarily ResNet-20 and VGG-11) on image classification tasks. Specifically, the method temporarily reduces the internal spatial resolution of feature maps via adaptive max-pooling inserted after an early residual stage, and progressively restores the native resolution before training concludes. Because the model returns to its standard resolution and architecture, inference cost and parameter count are completely unchanged.

The authors evaluate this intervention across various unaugmented experimental settings covering multiple datasets (CIFAR-10, SVHN, STL-10), architectures, optimizers (SGD, Adam, AdamW, RAdam), activations, normalizations, and budgets. In these settings without data augmentation, the method yields consistent mean test accuracy improvements ranging from 0.38 to 6.28 percentage points. The paper also introduces comparison points with Gaussian feature-map smoothing (G) and combined resolution/blur interventions (RG), and compares against prior methods including Curriculum by Smoothing (CBS) and SDPoint. Extensive controls analyze the necessity of coarse-to-fine ordering, single-resolution reductions, hyperparameter selection via held-out validation splits, training-loop wall-clock speedups, and loss landscape geometry.

Critical strengths
  • The authors clearly demonstrate and emphasize where the benefits diminish or vanish—specifically under standard strong data augmentations (random crops and horizontal flips) and in longer training regimes.

  • The authors investigate whether the coarse-to-fine schedule is essential by reversing or mixing resolution stages and by testing a single reduced resolution. These experiments provide evidence that temporary resolution reduction, rather than the particular progression, accounts for much of the improvement. Evaluation of normalization handling, showing how BatchNorm calibration conventions (saved statistics, clean recalibration, loader-matched momentum) alter baseline rankings, particularly when benchmarking against SDPoint and CBS.

  • The paper evaluates Hessian trace and leading eigenvalues via Lanczos iterations and filter-normalized random-direction sensitivities. The authors do not claim that lower loss curvature is a universal explanation for the gains, noting that reduced curvature in the reference setting does not hold as a universal mechanistic explanation across all optimizers (e.g., AdamW) or dataset splits.

  • The method is straightforward to implement and does not add trainable parameters or permanent inference-time computation. This makes it potentially useful in resource-constrained training settings.

Critical weaknesses

Major Weaknesses

  • Limited methodological novelty: The central operation is temporary adaptive max-pooling within a CNN, followed by restoration of native resolution. Although the authors distinguish their approach from SDPoint, CBS, and progressive resizing, the conceptual novelty over existing internal downsampling and curriculum-based methods is not sufficiently strong. The paper would benefit from a clearer account of what fundamental insight the proposed method contributes beyond an empirically effective variant of temporary downsampling.

  • The primary limitation is that modern deep learning pipelines for computer vision almost universally rely on data augmentation (e.g., random crops, flips etc) and extended training horizons. When basic crop-and-flip augmentation is added, the paired gain of the method over Plain drops drastically (e.g., from +3.25 down to +0.19 on CIFAR-10 at 60 epochs, and showing negligible or negative differences at 160 epochs depending on the BN calibration policy). While the paper notes that unaugmented regimes matter when domain-appropriate transformations are unavailable, the practical scope of the method remains largely restricted to data-scarce or unaugmented domains.

  • The empirical evaluation is restricted to small-scale convolutional models (ResNet-20 and VGG-11-BN) trained on relatively small image datasets (CIFAR-10, SVHN, STL-10). The paper contains no experiments on ImageNet-1k, modern ConvNets (e.g., ConvNeXt), or Vision Transformers (where patch/sequence reduction are present). This makes it difficult to ascertain whether internal spatial continuation scales to realistic computer vision pipelines.

  • While the paper does an good job showing that several hypotheses fail to explain the effect (the coarse-to-fine schedule order does not matter; Gaussian smoothing alone does not explain it; loss curvature reduction fails to track performance across optimizers), it does not provide much theoretical explaination of why temporary spatial downsampling acts as a regularizer.

Minor Weaknesses

  • While the authors transparently acknowledge using CIFAR-10 test set feedback to guide initial choices for reduction sites, schedules, and learning rates (Section 6, Appendix B), this iterative reuse introduces potential selection bias. Although the validation split experiment in Appendix G.3 partially mitigates this concern, its hyperparameter search remains limited, as both Plain and R selected the boundary learning rate of 0.04 without further expansion of the grid.

  • The sensitivity of relative performance rankings to the BatchNorm recalibration convention (P1 vs. P3) under augmented settings is significant (e.g., shifting differences between R and SDPoint by over 1 percentage point). While this is analyzed thoroughly, it indicates that empirical conclusions in these regimes are fragile.

  • The paper could more clearly specify when users should use temporary resolution reduction, when they should prefer standard training, and how to choose the reduction ratio and schedule without consulting test-set performance.

Questions for authors
  • Could you provide preliminary empirical evidence or analysis showing whether internal spatial continuation transfers to modern vision architectures—such as ConvNeXt or Vision Transformers (e.g., through token merging or downsampling)—beyond the smaller models (ResNet-20 and VGG-11-BN) evaluated in the paper?

  • Do the consistent improvements reported across CIFAR-10, SVHN, and STL-10 scale to more complex, large-scale benchmarks such as ImageNet-1k, where networks feature significantly deeper hierarchical representations?

  • Given the limited benefits observed under data augmentation, what specific applications or data domains are expected to benefit most from this intervention?

3 comments
AuthorsOpenReview ↗

Response to Reviewer xzV9 — 1/3: Imagenette / ResNet-18 transfer

We thank the reviewer for the thoughtful evaluation, including the recognition of our schedule ablations, BatchNorm controls, and careful separation of geometric observations from mechanistic explanations. We have run additional experiments aimed at the requested extensions to higher-resolution images, deeper convolutional models, and a modern ConvNet, and we can now be more specific about potential application domains.

1. Transfer to 128 × 128 images with ResNet-18.

To test whether the effect is restricted to the original 32 × 32 benchmarks and small ResNet-20 models, we evaluated R on Imagenette, a ten-class subset of ImageNet, with 128 × 128 inputs and ResNet-18. We considered both an unaugmented recipe and a recipe with input augmentation.

Protocol. Official imagenette2 320-px release, centre square crop and bilinear resize to 128, splits 8,469 / 1,000 (drawn from train) / 3,925 (official validation used as test); ResNet-18-BN; 30 epochs, effective batch 128, SGD (momentum 0.9, weight decay 5e-4), 60 warmup updates then cosine to zero. R pools at the first stage, to 16 then 24 then the native 32, at epochs 0 / 6 / 12, with the native network from epoch 13 on; the pooling is absent at inference. The augmented recipe adds zero padding 16, a uniform 128 crop and a horizontal flip with probability 1/2. The learning-rate grid 0.02-1.28 is shared by all methods and every selected rate is interior to it; selection on validation seeds 10-12, confirmation on test seeds 13-15, paired by seed, under saved BatchNorm statistics at the selected epoch. SDPoint is included under the same grid and rule. The protocol was frozen before any run was launched, and test accuracy was never consulted during selection.

Imagenette test accuracy (%)

Training recipePlain (mean ± SD)R (mean ± SD)SDPoint (mean ± SD)Paired R − Plain (pp)
No augmentation77.27 ± 0.3381.05 ± 0.5782.54 ± 0.66+3.79 ± 0.28 (3/3)
With augmentation84.08 ± 0.4684.57 ± 0.6882.68 ± 0.42+0.48 ± 0.78 (2/3)

The improvement of +3.79 ± 0.28 percentage points without augmentation, positive on 3 of 3 seeds, is encouraging evidence that the effect persists with a deeper network and substantially larger images. With augmentation, the mean advantage is smaller (+0.48 ± 0.78 points, positive on 2 of 3 seeds and therefore not resolved at this seed count; +1.03 ± 0.89, 3 of 3, after clean BatchNorm recalibration), consistent with the trend already documented in the submitted CIFAR-10 experiments. The takeaway is not that the same numerical gain holds universally; rather, the original relationship between augmentation strength and the additional value of resolution reduction also appears in this more demanding setting.

The augmented recipe also shows the clearest qualitative difference between the two interventions. SDPoint, selected under the same grid, reduces accuracy relative to Plain there (SDPoint − Plain = −1.41 ± 0.77 points, negative on 3 of 3 seeds), whereas R does not (R − SDPoint = +1.89 ± 1.07, positive on 3 of 3). Without augmentation the ordering is reversed (R − SDPoint = −1.49 ± 0.65, 0 of 3). We report both directions rather than either alone.

Imagenette is not ImageNet-1k, and we do not present it as such. It nonetheless directly addresses an intermediate, concrete scaling question. The phenomenon is not confined to ResNet-20 or 32 × 32 inputs.

AuthorsOpenReview ↗

Response to Reviewer xzV9 — 2/3: Preliminary ConvNeXt experiment

2. Preliminary ConvNeXt experiment.

We have additionally tested a ConvNeXt variant without BatchNorm, which helps probe whether the intervention can be applied beyond the original residual/VGG configurations and without relying on BatchNorm's running-statistics behavior.

Protocol. ConvNeXt-Femto (depths 2/2/6/2, widths 48/96/192/384, depthwise 7 x 7 convolutions, LayerNorm, 2 x 2 patchify stem) on CIFAR-10 with the same 45,000 / 5,000 / 10,000 split as above, no augmentation; 30 epochs, effective batch 128, AdamW with weight decay 0.05, 60 warmup updates then cosine to zero. R applies adaptive max pooling at the first stage, to 8 then 12 then the native 16, at epochs 0 / 6 / 12. The learning-rate grid 2.5e-4-0.032 is shared by all methods, extended twice by the frozen edge rule so that every selected rate is interior; selection on validation seeds 10-12, confirmation on test seeds 13-15, paired by seed. This network has no BatchNorm, so the three evaluation policies coincide.

MethodValidation-selected LRTest accuracy (mean ± SD)
Plain ConvNeXt0.016 (epoch 29)68.65 ± 3.49
ConvNeXt + R0.008 (epoch 23)76.62 ± 0.51
ConvNeXt + SDPoint0.008 (epoch 29)78.78 ± 0.35

The paired mean difference is +7.97 ± 3.33 percentage points, positive on 3 of 3 seeds. However, the Plain runs exhibit substantial variability (SD 3.49 against 0.51 for R; per-seed differences +9.80, +4.12, +9.98), so this numerical difference is not yet a stable estimate of the benefit. The selection rule sends Plain to 0.016, which is its best validation rate (72.55% against 67.93% at 0.008) but also the rate at which it is most seed-dependent, while R and SDPoint both select 0.008 and are stable there; SDPoint shows the same profile against Plain (+10.13 ± 3.83 points, 3 of 3). We include it as an exploratory compatibility result rather than a robust performance claim. It motivates the next scaling tests, but the stronger, presently interpretable transfer evidence is the ResNet-18/Imagenette experiment above. We have not tested Vision Transformers, for which changing the token sequence would introduce different design choices.

AuthorsOpenReview ↗

Response to Reviewer xzV9 — 3/3: Application domains and scientific contribution

3. Application domains and concrete guidance.

The submitted paper already shows that the benefit is strongest in no-augmentation settings, with substantial gains across twenty targeted combinations of datasets, architectures, optimizers, normalizations, and budgets. It also explicitly reports how conventional augmentation can reduce the extra gain. We view this as a useful characterization of the method's operating regime rather than a tension with the main conclusion.

An additional test on OrganAMNIST makes the application case more concrete. Its axial CT images include left- and right-specific class labels, so horizontal flipping is not automatically label-preserving. Under the validation-selected comparison described in our response to Reviewer pGXo, R improves on Plain by +0.61 ± 0.30 percentage points, positive on 3 of 3 seeds and under every evaluation policy, and has a higher mean accuracy than SDPoint (+0.68 ± 0.18 points, 3 of 3 under saved statistics; +0.15 ± 0.25 after recalibration). This does not exclude other medical-image augmentations; it illustrates why label-aware augmentation can require domain-specific decisions. A training-only internal intervention offers a complementary option without changing the input-label pairing.

For practitioners, we propose the following evidence-based guidance.

  • First consider R when training without augmentation or when appropriate augmentations are limited. This is where the original evidence is strongest, and the new OrganAMNIST and unaugmented Imagenette experiments broaden that case.
  • Expect smaller incremental gains when a strong augmented recipe is already effective. The original paper makes this dependence explicit, and the new augmented Imagenette result is in the same direction. R is a candidate to validate, not a guaranteed improvement.
  • A complex multi-stage schedule is not essential to test the underlying idea. Appendix G.2 shows that a single, moderately reduced side (24 rather than 32 in the reference ResNet-20 setting) retains most of R's gain, while stronger reduction need not help. The original schedule/site were selected in the CIFAR-10 exploration and carried into transfer settings with scale/budget adaptations; new applications should select adaptations on validation data, not test accuracy.
  • Inference remains exactly the original network. R introduces no extra inference-time operation or trainable parameters. Training time is recipe-dependent; the original timing results find it comparable in the short reference experiment and lower in the controlled long-SGD timing study, without claiming universal speedups.

4. The broader scientific contribution, and how SDPoint fits into it.

We agree with the reviewer's underlying request to identify an insight that transcends the particular implementation. Importantly, the paper already makes this distinction. The original motivation is spatial continuation, but our matched-exposure and single-resolution experiments demonstrate that temporary internal resolution reduction matters more than the monotone coarse-to-fine progression. Our result is therefore not simply that one hand-selected schedule works.

SDPoint is valuable in this interpretation. It reduces internal resolution through a different procedure—random, per-update average pooling at varying positions—whereas R uses deterministic, fixed-site max pooling. Yet both can improve the final model when evaluated at full resolution. Our additional stopped-SDPoint control (Appendix E.4) also shows that ending its random reductions when R returns to native resolution changes mean test accuracy by at most 0.16 points under the primary P1 BatchNorm evaluation policy in the three tested recipes. This is not a proof that all implementations are equivalent, but it supports studying training-time exposure to reduced internal spatial resolution as a shared empirical factor rather than attributing the effect to a single ordering rule.

Our contribution is precisely to analyze that factor. We separate reduction from progression, compare it with smoothing, document its dependence on augmentation and optimizer, and investigate geometric hypotheses without treating them as established explanations. The evidence supports an empirical generalization phenomenon, while leaving a complete mechanistic theory open. The new results broaden the architectural and domain scope of that account, rather than changing its original thesis.

We will include these additional experiments and make the already-present intervention-level conclusion more prominent in the revision, reporting every selection grid in full. We appreciate the reviewer identifying the most valuable extensions to test.

We would be happy to clarify any remaining questions during the discussion period.

Reviewer_ZiBtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies temporary reduction of internal feature-map resolution during CNN training. Its main procedure applies adaptive max-pooling at a fixed location, increases the spatial resolution during training, and ultimately restores the original architecture without additional inference cost. Experiments cover CIFAR-10, SVHN, and STL-10, with variations in architectures, optimizers, normalization, and training budgets. The method improves mean accuracy in all twenty reported settings without augmentation. Comparisons include Gaussian filtering, Curriculum by Smoothing, and SDPoint. Matched-exposure and single-resolution controls show that most of the benefit does not require a coarse-to-fine progression. The advantage substantially decreases with augmentation and stronger training recipes, while geometric diagnostics do not establish lower curvature as a general explanati

Critical strengths
  1. The schedule controls provide a useful empirical insight. Reversing or mixing the reduced-resolution epochs preserves the improvement, and a single moderate reduction retains most of it (Appendix G.1–G.2). These experiments distinguish the benefit of temporary resolution reduction from the motivating coarse-to-fine curriculum, making the study more informative than a performance comparison alone.

  2. The evaluation carefully documents important limitations. The paper reports paired comparisons, adverse results under augmentation, stronger SDPoint results in some recipes, and sensitivity to BatchNorm calibration. It also avoids treating reduced curvature as a demonstrated mechanism. This transparency supports the credibility of the narrower empirical conclusions.

Critical weaknesses
  1. The incremental contribution and practical advantage over existing spatial interventions remain limited. In the matched 60-epoch experiment, ordinary crop-and-flip augmentation reduces R's gain from 3.25 to 0.19 percentage points (Tables 17–18). In the 160-epoch unaugmented AdamW comparison, SDPoint exceeds R by 3.11 points (Table 24). The schedule ablations are informative, but they also show that the proposed progression contributes relatively little. Although the paper appropriately acknowledges these results, the evidence does not yet establish a sufficiently compelling advantage or broadly informative principle beyond existing downsampling-based regularization.

  2. The evidence remains partly exploratory, and the comparison with adequately tuned baselines is incomplete. Method selection and several learning-rate decisions used CIFAR-10 test results (Appendices B and C.2). The subsequent validation-based control is valuable and retains a 2.49 ± 0.24 point gain, but both methods select the largest tested learning rate, 0.04 (Appendix G.3). Thus, this control does not resolve whether the advantage persists after a sufficiently broad, comparable tuning procedure. Additional random seeds do not remove the earlier test-set exposure, so the twenty-setting result should not be interpreted as twenty independent confirmations.

Questions for authors
  1. Additional evidence relevant to the baseline concern: Do you have validation-selected results extending the learning-rate search beyond 0.04, with comparable tuning budgets for Plain and R? Both methods improve throughout the current grid. Establishing whether the remaining gain survives a wider search would materially affect my assessment.

  2. Additional evidence relevant to the contribution: Are validation-selected comparisons against SDPoint available on SVHN or STL-10, where the transfer experiments report benefits for R? The current direct comparisons are confined to CIFAR-10. Evidence of a clear advantage in a setting central to the paper's intended use would strengthen the case for the proposed intervention.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZiBt — 1/3: Extended learning-rate search

We thank the reviewer for the careful reading and constructive questions. We particularly appreciate the recognition of the matched-exposure and single-resolution controls, which are central to the paper's contribution. We have performed additional experiments directly addressing both requests, extending the validation-selected learning-rate search and comparing R with SDPoint on SVHN and STL-10.

1. Does R's improvement survive a broader learning-rate search?

The reviewer correctly identifies an important question about the validation-based control in Appendix G.3, where both Plain and R selected learning rates at the upper boundary of the tested grid (0.04). We have now expanded the CIFAR-10 search up to 0.64, while retaining a comparable tuning procedure for the two methods. Specifically, both methods receive the same candidate learning rates and selection budget; each independently selects its own configuration using held-out validation accuracy; final test evaluation uses separate seeds after the choices are fixed.

Protocol. CIFAR-10, split 45,000 / 5,000 / 10,000; ResNet-20-BN; 30 epochs, effective batch 128, SGD (momentum 0.9, weight decay 5e-4), 60 warmup updates then cosine to zero; no augmentation. The learning-rate grid 0.0025-0.64 is shared by both methods. For each method we take the (rate, completed epoch) maximising the mean validation accuracy over selection seeds 10-12, with saved BatchNorm statistics on the native network; test is then measured on confirmation seeds 13-15 at that rate and epoch, paired by seed (same initial weights, same data order). The protocol was frozen before any run was launched, and test accuracy was never consulted during selection.

Extended-grid CIFAR-10 results (test accuracy, %)

MethodValidation-selected LRTest accuracy (mean ± SD)
Plain0.32 (epoch 28)85.96 ± 0.15
R0.16 (epoch 30)87.09 ± 0.60

Both optima are interior to the grid, so the selection is no longer truncated for either method.

The paired test gain (R minus Plain) is +1.13 ± 0.75 percentage points, positive on 3 of 3 seeds, and it is stable across BatchNorm evaluation policies (+1.15 ± 0.76 after clean recalibration, +1.18 ± 0.67 under an EMA over the training loader). The mean gain is smaller than the +2.49 ± 0.24 points reported with the previous grid ending at 0.04. The arithmetic of that reduction is visible in the validation grid: Plain gains 3.63 validation points between 0.04 and 0.32, R only 1.90 between 0.04 and 0.16, so the earlier figure was inflated by the truncation, asymmetrically against the baseline. This is an important refinement of the magnitude, but the advantage in mean accuracy remains positive after expanding the common search range. This is precisely the additional control requested. The effect observed in the earlier validation experiment is not solely a consequence of stopping the search at 0.04. We will include the full selection protocol and per-method selected hyperparameters so that the comparison can be assessed independently of the headline gain.

AuthorsOpenReview ↗

Response to Reviewer ZiBt — 2/3: SDPoint on SVHN and STL-10

2. How does R compare with SDPoint on SVHN and STL-10?

We also extended the direct comparison to both datasets, with Plain, R, and SDPoint evaluated under the same candidate learning-rate grid and selection budget, with method-specific validation selection and separate test seeds.

Protocol. Same recipe, same selection rule and the same seeds as above. SVHN: 50,000 / 5,000 / official test, 30 epochs, grid 0.005-1.28 after the two grid extensions allowed by the frozen edge rule. STL-10: 4,000 / 1,000 / official test, 60 epochs, grid 0.005-0.64 after one extension. Neither uses augmentation. SDPoint is the paper's comparator: one stochastic point per residual block, ratios {0.5, 0.75}, full-resolution inference, and it is selected on validation under the same grid as the other two. Selected rates: SVHN Plain 0.32, R 0.64, SDPoint 0.08; STL-10 all three 0.32. Test accuracy is reported under P0 (saved BatchNorm statistics) at the selected epoch, with P1 (clean recalibration on the fitting set) quoted where it changes the reading.

Test results (accuracy, %); paired differences in points, with the number of seeds on which they are positive

DatasetPlain (mean ± SD)R (mean ± SD)SDPoint (mean ± SD)R − Plain (pp)R − SDPoint (pp)
SVHN95.66 ± 0.0895.73 ± 0.0296.06 ± 0.11+0.06 ± 0.07 (2/3)−0.33 ± 0.09 (0/3)
STL-1064.38 ± 1.0071.82 ± 1.1770.83 ± 2.80+7.44 ± 2.16 (3/3)+0.99 ± 3.32 (1/3)

On SVHN, R and Plain are essentially tied in mean accuracy (+0.06 ± 0.07 points for R, positive on 2 of 3 seeds), and SDPoint performs at least as well as R (R − SDPoint = −0.33 ± 0.09, 0 of 3; −0.78 ± 0.13 after recalibration). For reference, at the fixed rate of 0.005 used in the paper's transfer table, Plain reached 93.10 and R 94.26; once tuned, Plain reaches 95.66, so part of the gain reported there was a tuning artefact of the baseline. On STL-10, by contrast, R retains a substantial improvement over Plain (+7.44 ± 2.16 points, positive on 3 of 3 seeds), while SDPoint matches R's mean accuracy under saved statistics and exceeds it after clean BatchNorm recalibration: R − SDPoint is +0.99 ± 3.32 points (positive on only 1 of 3 seeds) under P0 and −2.48 ± 0.79 (0 of 3) under P1. The R-versus-Plain gains are resolved across seeds in both datasets' directions; the R-versus-SDPoint differences are not, except under recalibration, where SDPoint is ahead. STL-10 has a much smaller labeled training set than SVHN, making this contrast informative about the dependence of the benefit on the learning setting; it does not, by itself, establish a causal explanation.

The submitted paper already reports a regime in which R is ahead of SDPoint (the 30-epoch CIFAR-10 SGD comparison, with +0.63 ± 0.19 points under P1), alongside regimes where SDPoint is stronger. We therefore interpret the new results as further evidence of recipe-dependent rankings, not as a universal ranking between two methods. They also extend the paper's original observation that substantial improvements can arise from training with reduced internal spatial resolution, although their magnitude and the best implementation depend on the recipe. In particular, the STL-10 results suggest that the phenomenon is not confined to CIFAR-10, while the SVHN result identifies a setting in which additional tuning largely closes R's advantage over Plain.

AuthorsOpenReview ↗

Response to Reviewer ZiBt — 3/3: Contribution relative to SDPoint

3. Contribution relative to SDPoint as a complementary empirical investigation.

We would like to clarify a point that is already present in the submitted manuscript (Abstract, Secs. 1 and 6, Appendices G.1–G.2). The contribution is not a claim that one particular deterministic schedule is the best form of downsampling. Rather, our experiments distinguish the influence of temporary internal resolution reduction from the details of the curriculum used to implement it.

This distinction matters when interpreting the relationship to SDPoint. SDPoint applies randomized, per-update average-pooling reductions at varying locations; R uses a fixed-site max-pooling intervention and a deterministic schedule, and restores the original architecture before inference. Our evaluated SDPoint instance also uses full-resolution inference. Their different outcomes therefore help probe a common, broader training-time intervention rather than merely decide which schedule wins on a single benchmark.

Two controls in the original submission are particularly relevant.

  • Order and progression controls (Appendices G.1–G.2). Reversing or mixing the reduced-resolution epochs preserves the improvement; a single moderate reduced resolution captures most of it. This isolates an effect that does not require a monotone coarse-to-fine curriculum.
  • Stopped-SDPoint control (Appendix E.4, Table 26). Stopping SDPoint's random reductions when R returns to native resolution changes its mean accuracy by at most 0.16 percentage points under the primary P1 BatchNorm evaluation policy in the three tested recipes. Although this is not an equivalence proof and depends on the evaluation convention, it provides a useful cross-method indication that continuing stochastic reductions throughout training is not necessary for the observed gains in those settings.

Thus, the comparison with SDPoint is scientifically complementary as well as competitive. Performance improvements across different implementations motivate the shared question, while our ablations identify which ingredient is consequential and which apparently attractive curriculum choices are not essential. The Gaussian controls and the geometry analysis further rule out overly simple interpretations; we do not claim they establish a complete mechanism.

We will foreground this intervention-level contribution and add the new validation-selected results to the revision, with every selection grid reported in full. The original study already documented where augmentation, optimizer choice, and BatchNorm treatment affect outcomes; these additional tests strengthen its central conclusion with the more stringent tuning and cross-dataset comparisons requested by the reviewer. We thank the reviewer for the suggestions, which directly improve the evidence supporting that conclusion.

We would be happy to clarify any remaining questions during the discussion period.

Thank you for the additional experiments, which address my concern about the learning-rate search. However, my concern about the limited contribution beyond SDPoint remains, so I will keep my score of 2.

Reviewer_pGXoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

During the first 40% of training, the paper inserts a max-pooling layer after the second residual block of a ResNet-20, which shrinks the feature maps from 32x32 to 16x16. The size is then restored to 24 and 32, and the pooling layer is removed. On a small dataset, CIFAR-10 without data augmentation, test accuracy rises significantly ~4-5%. However, with random crop and flip, the gain became insignificant.

Critical strengths

The experiments are carefully controlled, with seed-paired comparisons over multiple settings and a validation protocol for choosing the learning rate and stopping epoch.

Critical weaknesses

The text is hard to read. Most sentences carry a qualifier, and the main claim is difficult to locate. Plus that the idea is incremental and there are a lot of non-informative trivality reporting, being wrapped as "being candid". This work seems to be an unfinished attempt from a pipeline of LLM-based research.

The general approach is OK with me, but the way it is presented and the current quality is below respectable forums.

Questions for authors

The paper argues that the approach is useful where suitable augmentation is unavailable. Can you show the gain on one real dataset of this kind?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pGXo — 1/2: OrganAMNIST experiment

We thank the reviewer for the feedback and for noting the care taken with paired comparisons and validation-based selection. We understand the request for a sharper central message and, especially, for a concrete dataset in which standard image augmentation cannot be applied without considering the label semantics. We have addressed the latter with an additional medical-imaging experiment and will make the former more prominent in the revision.

1. New real-world experiment on OrganAMNIST.

We evaluated R on OrganAMNIST, a medical image-classification dataset containing axial CT slices of anatomical structures. Crucially, the classification task distinguishes left- and right-sided structures. Therefore, a conventional horizontal flip cannot simply be used as a label-preserving transformation. Without appropriate label handling, it can change the meaning of the class. This is a concrete example of the practical issue raised in the review. We do not claim that all augmentation is impossible for OrganAMNIST—other task-appropriate transformations may be useful—but that common augmentation rules from natural-image classification are not automatically valid.

We compared standard training (Plain), temporary internal resolution reduction (R), and SDPoint, with each method receiving the same candidate learning rates, method-specific selection on held-out validation data, and evaluation on separate test seeds.

Protocol. MedMNIST+ OrganAMNIST at 64 x 64, official splits 34,561 / 6,491 / 17,778, 11 classes, grayscale; ResNet-20-BN with one input channel; 30 epochs, effective batch 128, SGD (momentum 0.9, weight decay 5e-4), 60 warmup updates then cosine to zero, no augmentation. The learning-rate grid 0.02-0.64 is shared by the three methods and every selected rate is interior to it. Selection: the (rate, completed epoch) maximising mean validation accuracy over seeds 10-12, saved BatchNorm statistics, native network; confirmation on test seeds 13-15, paired by seed. SDPoint: one stochastic point per residual block, ratios {0.5, 0.75}, full-resolution inference. The protocol was frozen before any run was launched, and test accuracy was never consulted during selection.

OrganAMNIST test accuracy (%)

MethodValidation-selected LRAccuracy (mean ± SD)
Plain0.04 (epoch 29)96.07 ± 0.31
R0.08 (epoch 29)96.67 ± 0.27
SDPoint0.08 (epoch 29)95.99 ± 0.19

R improves on Plain by +0.61 ± 0.30 percentage points, positive on 3 of 3 seeds, and the improvement holds under every evaluation policy we report (+0.65 ± 0.07 after clean recalibration, +0.90 ± 0.17 under an EMA). R also achieves a higher mean accuracy than SDPoint: +0.68 ± 0.18 points, positive on 3 of 3 seeds under saved statistics, +0.15 ± 0.25 (2 of 3) after recalibration. We report the full numbers rather than infer a reliable ranking from the mean alone: the second margin is descriptive only, and the three methods lie within 0.2 points of one another on validation (99.30, 99.34, 99.29), so the selection itself is narrow.

This result adds a domain-relevant test beyond the natural-image benchmarks in the submission. In particular, the method changes internal feature-map resolution during training, without introducing potentially label-changing transformations of the input data. It therefore provides a useful option to investigate when augmentation must be carefully constrained. As with the original experiments, the extra pooling operation is absent at inference, and the evaluated network has the same architecture and inference computation as Plain.

AuthorsOpenReview ↗

Response to Reviewer pGXo — 2/2: Scientific contribution and presentation

2. The contribution is an empirical insight, not merely another pooling schedule.

We would also like to make the paper's existing central claim more visible. The Abstract and Discussion already state the key finding. Across the twenty reported no-augmentation settings, temporary internal resolution reduction improves mean test accuracy by 0.38–6.28 points, while the final network remains unchanged. That range was measured at the fixed learning rates of the original recipes; under the per-method validation selection described above, the gain holds in full on STL-10 (+7.44 ± 2.16 points, 3 of 3 seeds), is smaller on CIFAR-10 (+1.13 ± 0.75) and essentially vanishes on SVHN (+0.06 ± 0.07). The revision will report the selected numbers alongside the fixed-rate range. More importantly, the schedule ablations show that most of this effect comes from temporary reduction itself, not from the monotone coarse-to-fine ordering initially used to motivate the study.

We believe this is a useful distinction. A method can be motivated by continuation without the continuation order ultimately being the crucial ingredient. The reversed/mixed-order controls and the single-resolution controls establish this experimentally rather than assuming it. The paper also tests whether Gaussian feature-map smoothing or measured local curvature can serve as simple explanations and finds that neither accounts for all of the observations.

Our comparison to SDPoint fits this same perspective. SDPoint is an important related method, and its success provides additional motivation for treating reduced internal resolution as an intervention worth studying. Our contribution is to systematically characterize that intervention, identify when its gain persists, and disentangle the temporary reduction from the specific schedule. That insight remains relevant even when a different implementation attains higher accuracy under a particular recipe; it is not predicated on R beating every existing alternative.

3. Making the main result easier to locate.

We will revise the presentation to lead with a short, direct statement of the empirical conclusion and its domain of applicability; organize the principal experiments around the questions of effectiveness, schedule necessity, and comparison with related interventions; and keep secondary diagnostics from obscuring the primary message. This is an improvement in emphasis and organization rather than a change of scientific claim. The distinction between temporary reduction and coarse-to-fine progression is already explicitly supported in the submitted paper.

In short, the new OrganAMNIST experiment provides the requested concrete application, while the clarified presentation will make the paper's original empirical contribution more immediately legible. Temporary reduction of internal spatial resolution is a simple, training-only intervention with demonstrated benefits in several settings where standard augmentation is absent or constrained, and the particular continuation schedule is not the main source of the effect.

We would be happy to clarify any remaining questions during the discussion period.

AuthorsOpenReview ↗

Response to Reviewer pGXo – about clarity and AI usage

Dear Reviewer,

We would like to come back to two points in your review: the difficulty of identifying our main contribution and your concern about the use of generative AI.

1. What we are actually trying to show

Looking again at our Introduction, we can see why our main point might be missed. We introduce the paper through continuation methods and spend quite some time explaining the intuition behind coarse-to-fine training. This was our starting motivation, but it is not what we consider the main result of the study.

What we found is more general: reducing the internal spatial resolution of a CNN during training can improve its generalization, and this benefit does not require a particular coarse-to-fine schedule.

The ablation experiments are important here. Reversing the order of reduced-resolution phases, mixing them, or using a single moderately reduced resolution retains much of the improvement. The progressive schedule was therefore useful for investigating the idea, but the reduction itself appears to be the important ingredient.

This is also why we compare our approach with SDPoint. SDPoint uses stochastic internal downsampling rather than our deterministic schedule. We do not claim that our particular schedule is generally better. In fact, our results show that the ranking between the methods depends on the training recipe. What interests us is the broader observation that training with reduced internal resolution can be beneficial, even when the reduction is implemented in different ways.

2. Where the presentation can be improved

We went back through the Introduction and found several places where we could be much more direct.

For example, the opening discussion of continuation methods gives the impression that we are primarily proposing a progressive optimization procedure. We will change this so that the paper first introduces the question of internal resolution reduction and its effect on generalization. Continuation will remain as motivation, but not as the starting point of the argument.

Some sentences are also unnecessarily abstract. “We study the family obtained from that handle” does not tell the reader clearly what we are doing. We can simply explain that we temporarily reduce the resolution of internal feature maps during training and restore the original network before inference.

Similarly, our discussion of learning large-scale structures before finer details could be read as an explanation of the observed gains. It is only an initial intuition, and our experiments do not establish it as the mechanism. We will make that distinction explicit.

Finally, expressions such as “these additional experiments locate the effect rather than merely bound it” are not sufficiently informative. We should say what the comparisons show: which training settings benefit from resolution reduction, where the gains become smaller, and what the controls allow us to conclude.

We will make corresponding changes to the Abstract, putting the empirical finding and its limitations before the secondary analyses.

3. About our use of generative AI

We also want to clarify how we used AI when preparing the paper.

As mentioned in our AI Use Statement, we used generative AI throughout parts of the research and writing process, including assistance with code, feedback on experimental protocols, drafting, and language editing.

For the Abstract and Introduction, AI was used during drafting and revision to explore alternative formulations, improve the organization of the text, and rework passages that were difficult to follow. This went beyond simple spelling or grammar correction. We reviewed and revised the resulting text ourselves, checking that the wording reflected what we had actually tested and observed.

We made the decisions about the research questions, experimental design, controls, and interpretation of the results. We also checked the reported numbers against our experimental records. AI helped us with the work, including how parts of it were expressed, but we remained responsible for the scientific decisions and the final manuscript.

We understand, however, that this does not resolve the concern about readability. If some passages make a relatively simple empirical finding sound more complicated than it is, then we need to rewrite them. We agree that the Abstract and Introduction can be clearer, and the changes described above are intended to address that problem directly.

Thank you for pointing this out. We hope these clarifications make both our contribution and our writing process easier to understand.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.