- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces class-count rotations, which reassign class training counts across runs to jointly estimate a shared count response and class-specific difficulty offsets. The authors observe that difficulty vectors across learning protocols are often approximately related by affine transformations. Based on this observation, they propose PACE, which reuses a source difficulty vector and calibrates it with one target training run to forecast per-class losses under unseen allocations. The paper provides identifiability and local portability results, with experiments spanning vision, text classification, language modeling, and regression.
Critical strengths
- The paper addresses a clear and practically relevant problem: reducing the cost of repeatedly measuring class-conditional learning curves for new models.
- The prospective experiments are compelling. On the two locked vision targets, PACE outperforms count-only prediction on all 28 held-out runs, reducing RMSE by 39.1% and 22.7%, respectively.
- The evaluation includes connectivity diagnostics, matched-budget comparisons, measured computational costs, and failure cases. The authors are transparent about the boundaries of their claims.
Critical weaknesses
- The additive model primarily considers each class’s own training count, without directly testing how other classes’ allocations affect its loss. The appendix also reports significant class-specific slope variation, suggesting that the estimated difficulty offsets may depend on the count range and rotation design.
- The symmetry-based theory provides a sufficient mechanism for expected losses near a particular synthetic task. Without experiments directly testing this mechanism, its explanatory relevance to natural datasets remains uncertain.
- PACE’s advantage over stronger baselines is not consistent. The target pin performs better on WRN and ACS and matches PACE on the PovertyMap confirmation study, where prediction fusion performs better still. Improvements over count-only prediction therefore do not establish general superiority over competitive alternatives.
- The breadth of independent evidence is more limited than the number of protocol pairs suggests: 55 of the 75 pairs come from a single seven-domain language-modeling task. Furthermore, the reported uncertainty does not fully capture variation from repeating source measurement and target calibration.
Questions for authors
- If a class’s own training count is fixed while the allocation among other classes changes, how stable is its test loss?
- Does recovery from a small number of rotations remain similarly accurate when evaluated against a completely independent reference estimate?
- Can a source–target fusion weight be selected without held-out outcomes, and how would this approach compare with PACE and the target pin?
- Do the performance gains remain stable when the entire source-measurement and target-calibration procedure is repeated?
4 comments
Weakness 2. The symmetry-based theory gives a sufficient mechanism near a synthetic task; without experiments that test this mechanism, its relevance to natural data is uncertain.
We tested the mechanism directly in two experiments. On the task of the theorem the predictions hold, and they fail when the symmetry assumption is broken. On natural data (CIFAR-100) the predicted effect is present, significantly in 8 of 9 pre-specified tests, but only in part, which is expected since natural classes are not symmetric. The revised paper therefore presents the theorem as one sufficient mechanism and does not claim that it explains portability on natural data.
Theorem 2 predicts the following: if a protocol treats all classes alike, making some classes slightly harder changes the class losses in the same pattern under every such protocol, up to one scale factor per protocol.
Experiment 1: the task of the theorem (simulation). We start from a task in which all classes are equally hard and make each class harder or easier by a chosen amount. We train four protocols that treat all classes alike (softmax regression, two MLPs, a small Transformer) and two that do not (class-specific loss weights, class-specific label smoothing).
| Prediction of Theorem 2 | Protocols that treat classes alike | Protocols that do not |
|---|---|---|
| Before any change, all classes have the same loss | holds | fails: losses differ by 0.11–0.43 |
| The losses change in the pattern of the change we made | correlation at least 0.999 | 10–35% of the change goes elsewhere |
| The remaining part shrinks with the square of the size of the change | exponent 2.04–2.26 in six of seven cases, 2.55 in the seventh | exponent 1.2–1.4 |
| Two protocols differ only by one scale factor | to within 1% | not applicable |
Experiment 2: natural data (CIFAR-100, 126 runs). We make chosen classes harder by replacing a share of their images with low-resolution copies, in training and in test. The shares follow one fixed pattern across classes, at three strengths (0.1, 0.2, 0.4). The prediction is that the class losses of each protocol change in that pattern, with a protocol-specific strength of response.
| Protocol | Correlation between the change in class losses and the pattern, at strength 0.1 / 0.2 / 0.4 | Strength of response, at 0.1 / 0.2 / 0.4 |
|---|---|---|
| ResNet-18, different training recipe | 0.19 / 0.35 / 0.44 | 0.16 / 0.16 / 0.16 |
| ResNet-18 | 0.28 / 0.26 / 0.40 | 0.44 / 0.27 / 0.26 |
| ResNet-34 | 0.51 / 0.63 / 0.65 | 0.47 / 0.44 / 0.40 |
- Eight of the nine correlations are significant (permutation test, ; the ninth has ).
- The effect is partial: the correlations are 0.19–0.65, far from the 0.999 of the simulation. Real classes differ in how much low resolution hurts them, which the task of the theorem rules out by construction.
Changes in the revised PDF:
- Section 3, after Theorem 2 (page 5): what the theorem does not show, and a pointer to these tests.
- Appendix C.10 (page 33, Table 31): the simulation and the CIFAR-100 experiment.
Weakness 3 and Question 3. PACE's advantage over the target pin is not consistent; can a source–target fusion weight be chosen without held-out outcomes, and how does it compare with PACE and the pin?
Yes, a fusion weight can be chosen without held-out outcomes. With one calibration run, the fused forecast outperforms the pin on WRN-28-10 and PovertyMap. On ACS, where the source transfers poorly, a second calibration run is needed, and the fused forecast then matches or outperforms the pin.
With one calibration run the weight is one half, i.e., the plain average of the PACE and pin forecasts. With two calibration runs it is estimated from how well the two runs agree on each class (Proposition 2). Both rules were fixed before the held-out runs of the new targets were trained. Entries are held-out RMSE (lower is better); targets where the pin outperforms or ties PACE are marked .
| Target | Pin (1 run) | PACE (1 run) | Fused (1 run) | Pin (2 runs) | Fused (2 runs) |
|---|---|---|---|---|---|
| Tiny-ImageNet ConvNeXt-T (paper) | 0.616 | 0.528 | 0.527 | – | – |
| CIFAR-100-LT WRN-28-10 (paper) | 0.679 | 0.724 | 0.610 | – | – |
| CIFAR-100-LT ViT-T (new) | 0.573 | 0.537 | 0.486 | 0.502 | 0.467 |
| Tiny-ImageNet ResNet-50 (new) | 0.689 | 0.613 | 0.578 | 0.582 | 0.552 |
| Food-101 ConvNeXt-T (new) | 0.643 | 0.527 | 0.525 | 0.526 | 0.476 |
| Language model Llama-style (new) | 0.265 | 0.237 | 0.235 | 0.248 | 0.253 |
| Language model GPT-style (new) | 0.325 | 0.278 | 0.298 | 0.260 | 0.242 |
| PovertyMap confirmation (paper) | 0.0503 | 0.0503 | 0.0445 | – | – |
| ACS (paper) | 0.100 | 0.319 | 0.166 | 0.083–0.086 | 0.082–0.085 |
| ACS random forest (new) | 0.085 | 0.516 | 0.268 | 0.0735 | 0.0735 |
| ACS MLP (new) | 0.353 | 0.661 | 0.406 | 0.306 | 0.298 |
(The forecasts for the new targets were locked before the held-out runs. The two-run entry for the ACS target of the paper was computed in hindsight.)
- One calibration run. The fused forecast outperforms the pin on every vision and language-model target (by 10% on WRN-28-10) and on PovertyMap (12%). It does not on ACS, where the source transfers poorly.
- Two calibration runs. The estimated weight detects that case: on ACS it puts 0.88–0.99 on the pin and matches or outperforms it. On the three new vision targets it outperforms the two-run pin by 5–10%.
- Where the fusion does not help. With one run it is 7% behind PACE on the GPT-style target, and with two runs it is 2% behind the pin on the Llama-style target.
Changes in the revised PDF:
- Section 5 (page 8): the comparison with the pin is stated per regime instead of "generally outperforms".
- Section 4 (page 6), Table 1 (page 8) and the paragraph discussing it (page 9): the two fusion rules and their results.
- Appendix C.8 (page 32, Table 30): the fused forecast across all studies.
- Appendix C.2 (pages 26–28): the hypotheses declared in advance and their outcomes.
Weakness 4 (first part). The independent evidence is narrower than the number of protocol pairs suggests, because 55 of the 75 pairs come from one language-modeling task.
We agree that the 75 protocol pairs are not 75 independent results, and the revised paper now reports the task-level figure next to that count. Counting each task once, the median cross-protocol correlation is 0.933 over the seven tasks and 0.916 without language modeling, and PACE outperforms count-only in every task that has a forecasting experiment.
| How the evidence is counted | Portability: cross-protocol correlation | Forecasting: PACE against count-only |
|---|---|---|
| Each task counted once (seven tasks) | median 0.933; mean 0.88 [0.79, 0.97] | wins in all 6 tasks with forecasting experiments; mean reduction 39% [24, 52] |
| Each task counted once, without language modeling | median 0.916; mean 0.87 [0.76, 0.95] | 35% |
The two lowest tasks are 20 Newsgroups (0.64) and ACS (0.78), as the paper reports.
Changes in the revised PDF:
- Section 5, portability paragraph (page 7): the task-level median next to the pair count.
- Appendix C.9, "Task-level accounting" (page 33).
Weakness 4 (second part) and Question 4. The reported uncertainty does not capture variation from repeating the source measurement and the target calibration; do the gains remain stable when the whole procedure is repeated?
Yes, the gains remain stable when the source measurement and the target calibration are both repeated with new seeds: over 20 end-to-end replications per target (5 retrained source pairs 4 calibration runs), the reduction is 39.1% 1.5 and 23.1% 1.2 (39.1% and 22.7% in the paper), and PACE wins every held-out run in all 40. We agree that the reported intervals did not capture this variation.
We repeated each part of the procedure on the two locked targets of the paper. Entries are RMSE reduction against count-only.
| What is repeated | Tiny-ImageNet ConvNeXt-T | CIFAR-100-LT WRN-28-10 |
|---|---|---|
| Nothing (paper; interval over held-out runs) | 39.1% [38.0, 40.3] | 22.7% [20.5, 25.1] |
| Calibration seed (same allocation, 3 new seeds) | 39.4%, 36.2%, 39.2% | 23.0%, 22.9%, 23.0% |
| Source runs (5 seeds) and calibration run (original and 3 new seeds), all 20 combinations; mean SD [min, max] | 39.1% 1.5 [36.1, 41.0] | 23.1% 1.2 [21.2, 24.4] |
| Source subset and calibration allocation, resampled from existing runs; median [95% interval] | 37.8% [34.9, 40.4] | 23.7% [19.2, 27.2] |
| Configurations in which PACE outperforms count-only | 20 of 20 and 364 of 364 | 20 of 20 and 300 of 300 |
- The comparison with the pin is unchanged: on WRN-28-10 the pin remains more accurate than PACE in all 20 replications (by 1.8–8.6%), and their average outperforms the pin in all 40.
- Resampling the source subset and the calibration allocation widens the intervals by a factor of 1.7–2.3.
- Over all 19 studies (8,494 resampled configurations), PACE outperforms count-only in every configuration in 18. The exception is 20 Newsgroups, the weakest case of the paper: 4.8% .
Changes in the revised PDF:
- Section 5, "Robustness to Training Noise" (page 8).
- Appendix C.3 (pages 28–29, Table 22): the end-to-end replications.
- Appendix C.4 (page 29, Table 23): the resampled intervals for all 19 studies.
Question 2. Does the recovery of the spectrum from a small number of rotations remain similarly accurate when it is evaluated against a completely independent reference?
Yes. Against an independent reference, the correlation of a two-rotation spectrum drops by a median of 0.016 across 36 protocol variants.
In the paper, the reference spectrum is fitted on all runs, including those in each small subset. We split the runs of each protocol into two disjoint halves, fitted the reference on one half, and drew the two-rotation subsets only from the other (up to 50 random splits per protocol).
| Protocol, two rotations | Reference shares runs (paper) | Independent reference | Independent, corrected for the noise of the half-size reference |
|---|---|---|---|
| Tiny-ImageNet ResNet-18 | 0.963 | 0.924 | 0.949 |
| CIFAR-100-LT ResNet-18 (imbalance factor 100) | 0.933 | 0.916 | 0.927 |
| Food-101 | 0.969 | 0.965 | 0.968 |
| ImageNet-LT | 0.927 | 0.906 | 0.923 |
| Language-model source | 0.942 | 0.935 | 0.940 |
| 20 Newsgroups CNN | 0.449 | 0.422 | 0.448 |
Most of the drop is the extra noise of a reference built from half as many runs (last column). 20 Newsgroups is weak under either reference and needs more than two rotations, as the paper already reports.
Changes in the revised PDF:
- Section 5, "Count Rotations Recover the Difficulty Spectrum" (page 7): the independent-reference result.
- Appendix C.7 and Table 29 (page 31): the analysis and the table above.
We hope this addresses your concerns, and we would be glad to clarify anything further.
We thank the reviewer for the constructive comments. We address each comment below, followed by the corresponding changes in the paper. In the revised PDF, changes made in response to your comments are in red, and changes shared with another reviewer are in violet.
At a fixed total budget, a class's test loss is stable up to a small measured term: the allocation of the other classes accounts for 2.5% of the variance of class difficulty (95% interval 0.9–4.5%), which is smaller than the seed-to-seed noise.
We ran this experiment on CIFAR-100-LT with ResNet-18. Twenty probe classes, one per superclass, keep the same training images in every run, so their own counts are fixed. Only the other 80 classes change, in three settings:
Values are cross-entropy loss. For scale, the SD of class difficulty across classes is 1.03.
Following the setup of the paper, all runs above have the same total budget. When the data of the other 80 classes is halved or doubled, a probe class's loss changes by −0.19 and +0.17, respectively.
Changes in the revised PDF:
The estimated offsets depend little on the count range or the rotation design: the spectrum of one protocol fitted on two different count ranges correlates at 0.956–0.982, and fitted under two different designs at 0.989. Class-specific slopes do vary, but PACE does not transfer them. It transfers only the offsets and fits the count response on the target, from the calibration run.
We refitted the spectrum on subsets of the existing runs that differ in count range or design.
Changes in the revised PDF: