acceptodds
Under review as a conference paper at ICLR 2027

Count-Controlled Class Difficulty Is Portable Across Learning Protocols

Abstract

Neural scaling laws characterize how average test loss changes as the amount of training data grows. They do not characterize how group-wise or class-wise losses vary when a fixed data budget is reallocated. Fitting separate scaling laws for each class is also statistically unstable and computationally expensive. Motivated by this, we introduce class-count rotations, which vary class counts across training runs and jointly recover a shared response to class count and a count-controlled class-difficulty spectrum. Surprisingly, difficulty spectra estimated independently across different learning protocols are approximately affine transformations of one another. Motivated by this finding, we propose PACE, which estimates this spectrum through controlled rotations on an inexpensive source model and uses a single calibration run on a more expensive target model to forecast its per-class losses under different class allocations. We establish when rotations identify the difficulty spectrum, provide a sufficient symmetry-based mechanism for its portability, and explain why a single run on the target system can be sufficient for calibration. Our experiments across vision, language, and regression show that the difficulty spectra can be recovered reliably. Moreover, after a reusable source measurement, PACE requires only one target calibration run yet achieves substantially lower per-class loss prediction RMSE than a baseline using only class training counts, outperforming it on every held-out run of the prospectively evaluated targets. These results show that class-conditional scaling behavior can be reused across learning protocols rather than remeasured from scratch.

Primary area:
learning theory
Keywords:
Scaling Laws, Data Mixtures, Transfer Learning, Class-Conditional Learning, Predictive Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EYHUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces class-count rotations, which reassign class training counts across runs to jointly estimate a shared count response and class-specific difficulty offsets. The authors observe that difficulty vectors across learning protocols are often approximately related by affine transformations. Based on this observation, they propose PACE, which reuses a source difficulty vector and calibrates it with one target training run to forecast per-class losses under unseen allocations. The paper provides identifiability and local portability results, with experiments spanning vision, text classification, language modeling, and regression.

Critical strengths
  1. The paper addresses a clear and practically relevant problem: reducing the cost of repeatedly measuring class-conditional learning curves for new models.
  2. The prospective experiments are compelling. On the two locked vision targets, PACE outperforms count-only prediction on all 28 held-out runs, reducing RMSE by 39.1% and 22.7%, respectively.
  3. The evaluation includes connectivity diagnostics, matched-budget comparisons, measured computational costs, and failure cases. The authors are transparent about the boundaries of their claims.
Critical weaknesses
  1. The additive model primarily considers each class’s own training count, without directly testing how other classes’ allocations affect its loss. The appendix also reports significant class-specific slope variation, suggesting that the estimated difficulty offsets may depend on the count range and rotation design.
  2. The symmetry-based theory provides a sufficient mechanism for expected losses near a particular synthetic task. Without experiments directly testing this mechanism, its explanatory relevance to natural datasets remains uncertain.
  3. PACE’s advantage over stronger baselines is not consistent. The target pin performs better on WRN and ACS and matches PACE on the PovertyMap confirmation study, where prediction fusion performs better still. Improvements over count-only prediction therefore do not establish general superiority over competitive alternatives.
  4. The breadth of independent evidence is more limited than the number of protocol pairs suggests: 55 of the 75 pairs come from a single seven-domain language-modeling task. Furthermore, the reported uncertainty does not fully capture variation from repeating source measurement and target calibration.
Questions for authors
  1. If a class’s own training count is fixed while the allocation among other classes changes, how stable is its test loss?
  2. Does recovery from a small number of rotations remain similarly accurate when evaluated against a completely independent reference estimate?
  3. Can a source–target fusion weight be selected without held-out outcomes, and how would this approach compare with PACE and the target pin?
  4. Do the performance gains remain stable when the entire source-measurement and target-calibration procedure is repeated?
4 comments

We thank the reviewer for the constructive comments. We address each comment below, followed by the corresponding changes in the paper. In the revised PDF, changes made in response to your comments are in red, and changes shared with another reviewer are in violet.


Weakness 1 (first part) and Question 1. The additive model considers only a class's own count; if that count is fixed while the allocation among the other classes changes, how stable is the class's test loss?

At a fixed total budget, a class's test loss is stable up to a small measured term: the allocation of the other classes accounts for 2.5% of the variance of class difficulty (95% interval 0.9–4.5%), which is smaller than the seed-to-seed noise.

We ran this experiment on CIFAR-100-LT with ResNet-18. Twenty probe classes, one per superclass, keep the same training images in every run, so their own counts are fixed. Only the other 80 classes change, in three settings:

  1. Seed only (10 runs): all counts are fixed and only the random seed changes.
  2. Random exchange (20 runs): the other 80 classes exchange their counts at random.
  3. Similar classes rich or poor (12 runs): the four classes that share the probe's superclass receive the largest counts in half of the runs and the smallest in the other half.
SettingRunsChange in a probe class's test loss
1. Seed only10SD across runs: 0.25
2. Random exchange20SD across runs: 0.30
3. Similar classes rich instead of poor12Mean loss higher by 0.16 [0.13, 0.20]

Values are cross-entropy loss. For scale, the SD of class difficulty across classes is 1.03.

  • Setting 2 adds an SD of 0.16 to setting 1, which is 2.5% of the variance of class difficulty.
  • In the most adverse setting (3), the loss rises by 0.16 on average and by 0.50 for the rarest probe classes (about 12 images).

Following the setup of the paper, all runs above have the same total budget. When the data of the other 80 classes is halved or doubled, a probe class's loss changes by −0.19-0.19 and +0.17+0.17, respectively.

Changes in the revised PDF:

  • Section 5.1, Limitations (page 10): the cross-class term and its size.
  • Appendix C.6 (page 31): the full experiment.

Weakness 1 (second part). Class-specific slopes vary, so the estimated difficulty offsets may depend on the count range and on the rotation design.

The estimated offsets depend little on the count range or the rotation design: the spectrum of one protocol fitted on two different count ranges correlates at 0.956–0.982, and fitted under two different designs at 0.989. Class-specific slopes do vary, but PACE does not transfer them. It transfers only the offsets and fits the count response on the target, from the calibration run.

We refitted the spectrum on subsets of the existing runs that differ in count range or design.

TestResult
Count range: imbalance factor 10 against 100, same protocol (CIFAR-100-LT, Tiny-ImageNet with two architectures, Food-101)r=0.956r = 0.956–0.982
Design: Latin-square rotations against random permutations (CIFAR-100-LT)r=0.989r = 0.989
Forecasting: gain of PACE over count-only for every combination of calibration range and held-out range (Tiny-ImageNet, CIFAR-100-LT)30–61%
Slopes against offsets: ResNet-34 and ViT-T on identical allocationsclass-slope deviations r=0.16r = 0.16; offsets r=0.91r = 0.91
  • Class slopes do differ (their SD is 11–25% of the common slope), but they account for only 0.8–2.5% of the loss variance on the vision tasks. Adding a class-slope term to PACE helped in 6 of the 19 studies and hurt in 13.
  • On 20 Newsgroups the two ranges correlate at 0.71–0.94, which is at the reliability ceiling of that task, so this case is inconclusive.

Changes in the revised PDF:

  • Appendix C.9, "Count range and design" (page 32).

Weakness 2. The symmetry-based theory gives a sufficient mechanism near a synthetic task; without experiments that test this mechanism, its relevance to natural data is uncertain.

We tested the mechanism directly in two experiments. On the task of the theorem the predictions hold, and they fail when the symmetry assumption is broken. On natural data (CIFAR-100) the predicted effect is present, significantly in 8 of 9 pre-specified tests, but only in part, which is expected since natural classes are not symmetric. The revised paper therefore presents the theorem as one sufficient mechanism and does not claim that it explains portability on natural data.

Theorem 2 predicts the following: if a protocol treats all classes alike, making some classes slightly harder changes the class losses in the same pattern under every such protocol, up to one scale factor per protocol.

Experiment 1: the task of the theorem (simulation). We start from a task in which all classes are equally hard and make each class harder or easier by a chosen amount. We train four protocols that treat all classes alike (softmax regression, two MLPs, a small Transformer) and two that do not (class-specific loss weights, class-specific label smoothing).

Prediction of Theorem 2Protocols that treat classes alikeProtocols that do not
Before any change, all classes have the same lossholdsfails: losses differ by 0.11–0.43
The losses change in the pattern of the change we madecorrelation at least 0.99910–35% of the change goes elsewhere
The remaining part shrinks with the square of the size of the changeexponent 2.04–2.26 in six of seven cases, 2.55 in the seventhexponent 1.2–1.4
Two protocols differ only by one scale factorto within 1%not applicable

Experiment 2: natural data (CIFAR-100, 126 runs). We make chosen classes harder by replacing a share of their images with low-resolution copies, in training and in test. The shares follow one fixed pattern across classes, at three strengths (0.1, 0.2, 0.4). The prediction is that the class losses of each protocol change in that pattern, with a protocol-specific strength of response.

ProtocolCorrelation between the change in class losses and the pattern, at strength 0.1 / 0.2 / 0.4Strength of response, at 0.1 / 0.2 / 0.4
ResNet-18, different training recipe0.19 / 0.35 / 0.440.16 / 0.16 / 0.16
ResNet-180.28 / 0.26 / 0.400.44 / 0.27 / 0.26
ResNet-340.51 / 0.63 / 0.650.47 / 0.44 / 0.40
  • Eight of the nine correlations are significant (permutation test, p<0.01p < 0.01; the ninth has p=0.06p = 0.06).
  • The effect is partial: the correlations are 0.19–0.65, far from the 0.999 of the simulation. Real classes differ in how much low resolution hurts them, which the task of the theorem rules out by construction.

Changes in the revised PDF:

  • Section 3, after Theorem 2 (page 5): what the theorem does not show, and a pointer to these tests.
  • Appendix C.10 (page 33, Table 31): the simulation and the CIFAR-100 experiment.

Weakness 3 and Question 3. PACE's advantage over the target pin is not consistent; can a source–target fusion weight be chosen without held-out outcomes, and how does it compare with PACE and the pin?

Yes, a fusion weight can be chosen without held-out outcomes. With one calibration run, the fused forecast outperforms the pin on WRN-28-10 and PovertyMap. On ACS, where the source transfers poorly, a second calibration run is needed, and the fused forecast then matches or outperforms the pin.

With one calibration run the weight is one half, i.e., the plain average of the PACE and pin forecasts. With two calibration runs it is estimated from how well the two runs agree on each class (Proposition 2). Both rules were fixed before the held-out runs of the new targets were trained. Entries are held-out RMSE (lower is better); targets where the pin outperforms or ties PACE are marked ◀\blacktriangleleft.

TargetPin (1 run)PACE (1 run)Fused (1 run)Pin (2 runs)Fused (2 runs)
Tiny-ImageNet →\to ConvNeXt-T (paper)0.6160.5280.527––
CIFAR-100-LT →\to WRN-28-10 (paper) ◀\blacktriangleleft0.6790.7240.610––
CIFAR-100-LT →\to ViT-T (new)0.5730.5370.4860.5020.467
Tiny-ImageNet →\to ResNet-50 (new)0.6890.6130.5780.5820.552
Food-101 →\to ConvNeXt-T (new)0.6430.5270.5250.5260.476
Language model →\to Llama-style (new)0.2650.2370.2350.2480.253
Language model →\to GPT-style (new)0.3250.2780.2980.2600.242
PovertyMap confirmation (paper) ◀\blacktriangleleft0.05030.05030.0445––
ACS (paper) ◀\blacktriangleleft0.1000.3190.1660.083–0.0860.082–0.085
ACS →\to random forest (new) ◀\blacktriangleleft0.0850.5160.2680.07350.0735
ACS →\to MLP (new) ◀\blacktriangleleft0.3530.6610.4060.3060.298

(The forecasts for the new targets were locked before the held-out runs. The two-run entry for the ACS target of the paper was computed in hindsight.)

  • One calibration run. The fused forecast outperforms the pin on every vision and language-model target (by 10% on WRN-28-10) and on PovertyMap (12%). It does not on ACS, where the source transfers poorly.
  • Two calibration runs. The estimated weight detects that case: on ACS it puts 0.88–0.99 on the pin and matches or outperforms it. On the three new vision targets it outperforms the two-run pin by 5–10%.
  • Where the fusion does not help. With one run it is 7% behind PACE on the GPT-style target, and with two runs it is 2% behind the pin on the Llama-style target.

Changes in the revised PDF:

  • Section 5 (page 8): the comparison with the pin is stated per regime instead of "generally outperforms".
  • Section 4 (page 6), Table 1 (page 8) and the paragraph discussing it (page 9): the two fusion rules and their results.
  • Appendix C.8 (page 32, Table 30): the fused forecast across all studies.
  • Appendix C.2 (pages 26–28): the hypotheses declared in advance and their outcomes.

Weakness 4 (first part). The independent evidence is narrower than the number of protocol pairs suggests, because 55 of the 75 pairs come from one language-modeling task.

We agree that the 75 protocol pairs are not 75 independent results, and the revised paper now reports the task-level figure next to that count. Counting each task once, the median cross-protocol correlation is 0.933 over the seven tasks and 0.916 without language modeling, and PACE outperforms count-only in every task that has a forecasting experiment.

How the evidence is countedPortability: cross-protocol correlationForecasting: PACE against count-only
Each task counted once (seven tasks)median 0.933; mean 0.88 [0.79, 0.97]wins in all 6 tasks with forecasting experiments; mean reduction 39% [24, 52]
Each task counted once, without language modelingmedian 0.916; mean 0.87 [0.76, 0.95]35%

The two lowest tasks are 20 Newsgroups (0.64) and ACS (0.78), as the paper reports.

Changes in the revised PDF:

  • Section 5, portability paragraph (page 7): the task-level median next to the pair count.
  • Appendix C.9, "Task-level accounting" (page 33).

Weakness 4 (second part) and Question 4. The reported uncertainty does not capture variation from repeating the source measurement and the target calibration; do the gains remain stable when the whole procedure is repeated?

Yes, the gains remain stable when the source measurement and the target calibration are both repeated with new seeds: over 20 end-to-end replications per target (5 retrained source pairs ×\times 4 calibration runs), the reduction is 39.1% ±\pm 1.5 and 23.1% ±\pm 1.2 (39.1% and 22.7% in the paper), and PACE wins every held-out run in all 40. We agree that the reported intervals did not capture this variation.

We repeated each part of the procedure on the two locked targets of the paper. Entries are RMSE reduction against count-only.

What is repeatedTiny-ImageNet →\to ConvNeXt-TCIFAR-100-LT →\to WRN-28-10
Nothing (paper; interval over held-out runs)39.1% [38.0, 40.3]22.7% [20.5, 25.1]
Calibration seed (same allocation, 3 new seeds)39.4%, 36.2%, 39.2%23.0%, 22.9%, 23.0%
Source runs (5 seeds) and calibration run (original and 3 new seeds), all 20 combinations; mean ±\pm SD [min, max]39.1% ±\pm 1.5 [36.1, 41.0]23.1% ±\pm 1.2 [21.2, 24.4]
Source subset and calibration allocation, resampled from existing runs; median [95% interval]37.8% [34.9, 40.4]23.7% [19.2, 27.2]
Configurations in which PACE outperforms count-only20 of 20 and 364 of 36420 of 20 and 300 of 300
  • The comparison with the pin is unchanged: on WRN-28-10 the pin remains more accurate than PACE in all 20 replications (by 1.8–8.6%), and their average outperforms the pin in all 40.
  • Resampling the source subset and the calibration allocation widens the intervals by a factor of 1.7–2.3.
  • Over all 19 studies (8,494 resampled configurations), PACE outperforms count-only in every configuration in 18. The exception is 20 Newsgroups, the weakest case of the paper: 4.8% [−1.2,12.3][-1.2, 12.3].

Changes in the revised PDF:

  • Section 5, "Robustness to Training Noise" (page 8).
  • Appendix C.3 (pages 28–29, Table 22): the end-to-end replications.
  • Appendix C.4 (page 29, Table 23): the resampled intervals for all 19 studies.

Question 2. Does the recovery of the spectrum from a small number of rotations remain similarly accurate when it is evaluated against a completely independent reference?

Yes. Against an independent reference, the correlation of a two-rotation spectrum drops by a median of 0.016 across 36 protocol variants.

In the paper, the reference spectrum is fitted on all runs, including those in each small subset. We split the runs of each protocol into two disjoint halves, fitted the reference on one half, and drew the two-rotation subsets only from the other (up to 50 random splits per protocol).

Protocol, two rotationsReference shares runs (paper)Independent referenceIndependent, corrected for the noise of the half-size reference
Tiny-ImageNet ResNet-180.9630.9240.949
CIFAR-100-LT ResNet-18 (imbalance factor 100)0.9330.9160.927
Food-1010.9690.9650.968
ImageNet-LT0.9270.9060.923
Language-model source0.9420.9350.940
20 Newsgroups CNN0.4490.4220.448

Most of the drop is the extra noise of a reference built from half as many runs (last column). 20 Newsgroups is weak under either reference and needs more than two rotations, as the paper already reports.

Changes in the revised PDF:

  • Section 5, "Count Rotations Recover the Difficulty Spectrum" (page 7): the independent-reference result.
  • Appendix C.7 and Table 29 (page 31): the analysis and the table above.

We hope this addresses your concerns, and we would be glad to clarify anything further.

Reviewer_hFB5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to explain the learning curve per class, without full retraining with varying amounts of classes.

Critical strengths

I think it is pretty clear that computing the learning curve per class is way to expensive. So the goal of the paper is very crystal clear, especially I like the explicit research question on the first page.

Critical weaknesses

The paper is not written in a clear way. Despite careful reading, and besides that the method is (I believe) quite simple, the paper is written in a very dense way.

Figure 1 is unfortunately very hard to parse. I didn't get much insight from it.

The writing is not precise and introduces difficult to understand terms. For example: "Repeating this measurement independently across architectures and training procedures on the same task reveals an interesting property: the spectra are largely portable across protocols after a shift and rescaling. This does not mean that numerical difficulty values or complete learning curves are identical across models; rather, much of a protocol’s difficulty spectrum can be recovered from a shared task-level direction through a protocol-specific location and scale calibration (Fig. 1, center)." I have read this multiple times, but to me this still remains really hard to grasp; the concept of portability.

Rotations in line 160 are not clearly defined, and could clearly use an example. The graph I could also not understand. Is the graph pertaining to a single run, or to multiple runs? What is KK?

"approximately portable accross protocols" (line 178). It is not clear what this means. Do you mean that, I fix my protocol, and then I change my class proportions? Or do you mean that, I have 2 protocols (e.g. different architectures) and that it "transfers" accross architectures? Are the proportions then fixed? Given I got stuck here, I also cannot understand the definitions of \mu, \gamma, and e, which are also quite unclearly explained and rather informal.

"reduced-strength signal for class" I have no clue what this means. There is also no citation to explain it. The whole example I could not understand absent from this definition.

It is not clear what assumptions are made in Theorem 2 from the main body. It would be good to give some ideas of what the assumptions are that were made.

I have the feeling that Theorem 1 and Theorem 3 are rather trivial, and more or less follow from a linearly solveable system of equations. Because of this reason, I strongly encourage the authors to rewrite this, to make it more clear and easy to understand. But correct me if I am wrong or missing something here.

Questions for authors

Line 221: why is it "inexpensive"?

Line 227: I get lost here with the definition of z. What is a "count feature"?

Doesnt Figure 2B illustrate that, with disconnected graphs, there is no problem? Because in fact the RMSE is quite low for the red dots and compareable to the blue ones?

In line 302, it is mentioned that you tackle "regression" datasets. But I thought you assumed that we are in a classification setting? So what are the class counts in a regression setting?

3 comments

We thank the reviewer for pointing out the exact places where the text was unclear. We address each comment below, followed by the corresponding changes in the paper. No method or result changed. In the revised PDF, changes made in response to your comments are in green.


Weaknesses 1 and 2. The paper is written in a dense way, and Figure 1 is hard to parse.

You are right that the method is simple and that our writing hid it. We rewrote the method section and redrew Figure 1. Section 3 now gives the definitions and a worked example with its table before any theorem, and Figure 1 shows the method in three steps:

  1. Measure. We train one model type a few times, each time giving the classes different numbers of training examples. Comparing two classes when they had the same number of examples shows which is harder, independent of count. This gives one difficulty number per class (the difficulty spectrum of the paper).
  2. Observe. Doing the same for a different model type, we find that its difficulty numbers are close to a shifted and rescaled copy of the first model's.
  3. Forecast. For a new model we reuse the difficulty numbers already measured and train the new model once, to fit the shift, the scale and the effect of count. We can then predict its per-class losses for class proportions that were never trained.

Changes in the revised PDF:

  • Figure 1 (page 2): redrawn.
  • Section 3 (pages 3–4): definitions first, then the worked example with its table (page 4).
  • Theorem 2 (page 5) and Theorem 4 (page 6): each is preceded by a one-sentence statement in words.
  • Abstract (page 1) and list of contributions (pages 2–3): reworded.
  • Terms are defined where they first appear (run, rotation design, KK, count feature, target pin).

Weaknesses 3 and 5. "Portable across protocols" is hard to grasp: is one protocol fixed while the class proportions change, or are there two protocols; and what are μ\mu, γ\gamma and ee?

Your second reading is the right one. A protocol is an architecture together with its training recipe (optimizer, learning-rate schedule, number of training steps, data augmentation), so ResNet-18 and ConvNeXt-T trained on the same task, with the same classes, are two different protocols. "Across protocols" means from one of them to the other.

The class proportions are not fixed. They change from run to run inside each protocol, and that is what lets us remove the count effect and keep one difficulty number per class for each protocol.

"Portable" means that the difficulty numbers measured on one protocol tell us those of another: the second protocol's numbers have the same structure, only shifted and rescaled. So if we plot one protocol's difficulty numbers against the other's, one point per class, the points lie close to a straight line. The three symbols describe that line:

  • μ\mu (one number) is where the line sits. It absorbs the two protocols' different overall loss levels, and is zero once both sets of difficulty numbers are centered.
  • γ\gamma (one number) is the slope of the line. If γ=0.5\gamma = 0.5, two classes that are 1 apart for the first protocol are 0.5 apart for the second.
  • ee (one number per class) is each point's distance from the line. Portability holds when these are small.

On Tiny-ImageNet, the difficulty numbers of ResNet-34, ViT-T and ConvNeXt-T each correlate with those of ResNet-18 at 0.93–0.95. The weakest task in the paper, 20 Newsgroups, is at 0.64 (Limitations, page 10).

Changes in the revised PDF:

  • Introduction (page 2): the passage you quoted is replaced by this description.
  • Section 3 (page 4): the definition of portability and the meaning of μ\mu, γ\gamma and ee.

Weakness 4. Rotations are not clearly defined and need an example; is the graph for a single run or for multiple runs, and what is KK?

Each individual training of one protocol is a run. A count rotation is a few runs of the same protocol. Every run uses the same counts, for example 100, 30 and 10 examples. What changes from run to run is which class gets which count. The whole collection is represented by a single graph (not one graph per run). The graph has a vertex for each class and a vertex for each count, and an edge joins a class to a count if the class received that count in at least one run. KK is the number of distinct counts, not the number of runs.

Here is the example now in Section 3 (page 4), with three classes and K=3K = 3 counts (100, 30, 10). Cells show count / loss; the numbers are invented and noise-free.

Class AClass BClass C
Run 1100 / 0.930 / 1.610 / 2.4
Run 230 / 1.310 / 2.1100 / 1.5
Run 310 / 1.8100 / 1.230 / 1.9
  • Why rotate. Inside Run 1, A's loss is 0.7 lower than B's, but A also had more data, so this mixes difficulty with count. At equal count (A in Run 1 and B in Run 3 both have 100), the difference is 0.9−1.2=−0.30.9 - 1.2 = -0.3. That is the difficulty difference with the count effect removed.
  • The graph. If the graph is in one piece, every pair of classes can be compared at equal count, directly or through a chain. Runs 1 and 2 together already give one piece. Run 1 alone gives three separate pieces, and no pair can be compared.

Changes in the revised PDF:

  • Section 3 (pages 3–4): definitions of run, rotation design, rotation graph and KK, followed by this example and its table (page 4).
  • Appendix C.1 (pages 26–27, Table 19 and Figure 5): the same example extended to four runs, with the graphs drawn, including a case where the graph is in two pieces.

Weaknesses 6 and 7. "Reduced-strength signal" is not explained, and the assumptions of Theorem 2 are not given in the main body.

Both were missing from the main text and are now in it. The theorem is about an artificial task that we built to show how portability can arise. We first build artificial data points that identify their class completely: each input is a fixed pattern that belongs to one class only. We then weaken some of them in a controlled way, by scaling the pattern down by a fixed factor, so that it is fainter and harder to learn. This is what we meant by "reduced-strength signal". We control how many examples of each class are weakened, and a class is harder when more of its examples are weakened.

The assumptions, now stated before the theorem:

  1. The training procedure treats all classes alike (no class-specific settings).
  2. Training is finite: a network of finite width, a finite number of steps, and a finite expected loss.
  3. The rotation runs are balanced: every class gets every count equally often, and every pair of classes gets every pair of counts equally often.

Under these assumptions, when classes differ only slightly in their share of weakened examples, every such training procedure produces the same pattern of class difficulty, up to a shift and a scale. The theorem is one sufficient mechanism in an artificial task and not a guarantee for natural data, and the text now says so.

Changes in the revised PDF:

  • Section 3, "Why Class Difficulty Can Be Portable" (pages 4–5): the task in plain words, the three assumptions, the theorem stated in one sentence before the formal version, and what it does not show.

Weakness 8. Theorems 1 and 3 seem rather trivial and follow from a linearly solvable system of equations.

You are right. Both are conditions for a linear system to have a unique solution, and Theorem 1 is a classical result on connected designs, which we now cite. We no longer present them as main results: the abstract, the introduction and the list of contributions no longer say that we "prove" or "establish" them, and each is introduced as an elementary identification condition. We keep them because they are useful in practice. Theorem 1 says which sets of runs can work (the graph must be in one piece), and Theorem 3 says which single target run can work.

The contribution of the paper does not rest on these two results. It is the empirical finding that count-controlled class difficulty carries over between protocols, and the forecasts built on it, tested on targets whose predictions were fixed before training. The other theoretical results are Theorem 2 (how portability can arise in an artificial task) and Theorem 4 (without a source, at least C−1C-1 target runs are needed to match what one run gives with it).

Changes in the revised PDF:

  • Abstract (page 1), introduction (page 2) and Contribution 3 (pages 2–3): reworded as above.
  • Section 3 (page 4) and Section 4 (page 5): Theorems 1 and 3 are each introduced as an elementary identification condition.

Question 1. Line 221: why is it "inexpensive"?

The word was too loose and we removed it. The source has a real cost, but it is measured once and reused for every new target. In vision, one source run costs between 1/32 and 1/5 of a target run. In language modeling, one source measurement served ten targets, for 86 GPU-hours in total against at least 241 without it.

Changes in the revised PDF:

  • Section 4 (page 5): "inexpensive" is replaced by a statement of the shared cost with these numbers.

Question 2. Line 227: what is a "count feature" zz?

It is a fixed function of a class's training count nn that we use to model how loss depends on count. In our main experiments it is simply log⁡n\log n: loss is modeled as a straight line in log⁡n\log n.

Changes in the revised PDF:

  • Section 4 (page 5): the definition is given where zz first appears.

Question 3. Does Figure 2B show that disconnected graphs are no problem, since the red points have RMSE as low as the blue ones?

No, and our figure invited that reading. For a disconnected graph the red points measure the error only for pairs of classes inside the same piece of the graph, which can be compared. Pairs in different pieces cannot be compared at all, so they have no finite error to plot. In the example above, Run 1 alone leaves A, B and C in three separate pieces: any difficulty values for them fit the data exactly, as long as the count effect is shifted to compensate.

Changes in the revised PDF:

  • Figure 2 (page 7): the caption says what the red points cover.
  • Appendix C.1 (page 28, Figure 6): Figure 2B redrawn with the two cases in separate panels.

Question 4. Line 302: what are the class counts in a regression setting?

There a "class" is a group of examples: a country in PovertyMap (16 groups) or a US state in ACS (48 groups). The count is the number of training examples from the group, and the per-class loss is the group's mean squared error on test data. Nothing else in the method changes.

Changes in the revised PDF:

  • Section 5 (page 6): stated in the experimental setup.

We hope the new Section 3 resolves the clarity concerns. If a definition is still unclear, or if a specific technical concern remains, we would be glad to clarify it further.

Reviewer_aCHPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether class-conditional test losses under data reallocation can be forecast across learning protocols. It models loss as a shared count response plus a class-specific offset and uses class-count rotations to identify the offsets. Its main empirical observation is that the resulting difficulty spectra are often approximately affinely related across protocols on the same task. PACE transfers a source spectrum and fits an intercept, a source-to-target scale, and a count response using the per-class losses from one target calibration run. The paper provides identification conditions, a local symmetry-based explanation of portability, a calibration-design criterion, and a stylized target-run comparison. Experiments span vision, text classification, language modeling, and regression, including reportedly locked prospective forecasts on two vision targets and additional forecasting studies.

Critical strengths
  • Separating count effects from persistent class effects and asking whether the latter transfer across protocols is a useful extension of aggregate learning curves. The central contribution is a measurable cross-protocol regularity with a concrete forecasting use, rather than a requirement to improve classifier accuracy directly.
  • The authors distinguish retrospective spectrum estimation from forecast inputs and label post-hoc comparisons. They acknowledge that target pin wins on WRN and ACS, that hierarchical slopes can improve forecasting, that source costs must be amortized, and that the 75 protocol pairs do not represent 75 independent tasks. Complete-run clustering and the explicit reporting of evaluation-set instability in PovertyMap are also welcome.
Critical weaknesses
  • The target comparisons establish that adding source information can help, but count-only has no persistent class information, and target pin has no source information.
  • The paper appropriately uses held-out training runs as the replication unit. However, held-out forecasts share a fitted source spectrum and a designated target calibration run. Bootstrap intervals over held-out runs alone condition on those shared measurements; they do not establish how performance varies when the source measurement or calibration training seed is repeated.
Questions for authors
  • Using the existing source outcomes, how does target forecasting change when the transferred direction is a one-run raw-loss or simply count-adjusted source vector rather than the rotation-estimated spectrum?
  • How much residual allocation dependence remains when a class's own count is fixed but other classes' counts change?
  • Are the reported confidence intervals conditional on a fixed source estimate and target calibration run? How sensitive are gains to alternative source subsets and calibration seeds or allocations?
2 comments

We thank the reviewer for the careful review. We address each comment below, followed by the corresponding changes in the paper. In the revised PDF, changes made in response to your comments are in blue, and changes shared with another reviewer are in violet.


Weakness 1 and Question 1. The baselines are not information-matched (count-only has no class information, the target pin has no source information); would a one-run source vector, raw or count-adjusted per-class loss, do as well as the rotation-estimated spectrum?

PACE is more accurate than both source vectors you named on all seven locked targets (the two of the submission and five new ones).

Each alternative replaces only the source vector of PACE. The calibration run, the fit and the held-out runs are unchanged, so every method has class information from the source and one target run:

  1. Raw source loss: the per-class loss of one source run.
  2. Count-adjusted source loss: the same loss after a log-linear count trend is removed.
  3. PACE: the spectrum from rotated source runs (two in vision, fourteen in language modeling).

The pin, which uses the target run only, is shown for reference. Entries are RMSE reduction against count-only; the best value in each row is in bold.

TargetPinRaw source loss, 1 runCount-adjusted source loss, 1 runPACE
Tiny-ImageNet →\to ConvNeXt-T (paper)29%13%33%39%
CIFAR-100-LT →\to WRN-28-10 (paper)27%5%20%23%
CIFAR-100-LT →\to ViT-T (new)23%2%26%28%
Tiny-ImageNet →\to ResNet-50 (new)33%12%35%41%
Food-101 →\to ConvNeXt-T (new)21%7%34.6%35.4%
Language model →\to Llama-style (new)58%48%41%62%
Language model →\to GPT-style (new)49%48%41%56%
  • Raw source loss gains only 2–13% in vision, against 23–41% for PACE. The gain of PACE is therefore not explained by class information alone.
  • Count-adjusted source loss is below PACE on all seven targets.
  • The pin is below PACE on six of the seven targets. WRN-28-10 is the exception, as the paper reports.

A count adjustment of one run is valid only if counts are unrelated to class difficulty, which rotations guarantee by design. To test this, we trained 12 source runs whose counts follow class difficulty and transferred each vector to WRN-28-10:

Source counts tied to difficultyRaw source loss, 1 runCount-adjusted source loss, 1 runPACE (its own rotated runs)
Hard classes rare, correlation ≈−0.5\approx -0.514–16%10–15%23%
Hard classes common, correlation ≈+0.5\approx +0.50–2%14–17%23%
Hard classes common, correlation ≈+0.97\approx +0.9718–19%0%23%
Hard classes rare, correlation ≈−0.97\approx -0.9726–27%0%23%

The gain of the adjustment falls from 20% to 10–17% and then to 0% as the counts follow difficulty more closely. Raw loss is ahead in one row, where the counts themselves carry the difficulty ranking.

Changes in the revised PDF:

  • Abstract (page 1) and Contribution 1 (page 2): rotations are presented as a design that assigns counts independently of class difficulty.
  • Section 5.1 (page 10): new Table 3, comparing the pin, the two one-run vectors and PACE on all seven locked targets and on source runs whose counts are tied to difficulty. The component-ablation table moved to Appendix C.5 (Table 25).
  • Appendix C.5 (pages 29–31, Tables 24 and 26–28) and Appendix C.2 (Table 21, page 28): the full comparisons.

Question 2. How much does a class's loss still depend on the allocation when its own count is fixed and the other classes' counts change?

At a fixed total budget, the allocation of the other classes accounts for 2.5% of the variance of class difficulty (95% interval 0.9–4.5%), which is smaller than the seed-to-seed noise.

We ran this experiment on CIFAR-100-LT with ResNet-18. Twenty probe classes, one per superclass, keep the same training images in every run, so their own counts are fixed. Only the other 80 classes change, in three settings:

  1. Seed only (10 runs): all counts are fixed and only the random seed changes.
  2. Random exchange (20 runs): the other 80 classes exchange their counts at random.
  3. Similar classes rich or poor (12 runs): the four classes that share the probe's superclass receive the largest counts in half of the runs and the smallest in the other half.
SettingRunsChange in a probe class's test loss
1. Seed only10SD across runs: 0.25
2. Random exchange20SD across runs: 0.30
3. Similar classes rich instead of poor12Mean loss higher by 0.16 [0.13, 0.20]

Values are cross-entropy loss. For scale, the SD of class difficulty across classes is 1.03.

  • Setting 2 adds an SD of 0.16 to setting 1, which is 2.5% of the variance of class difficulty.
  • In the most adverse setting (3), the loss rises by 0.16 on average and by 0.50 for the rarest probe classes (about 12 images).

Following the setup of the paper, all runs above have the same total budget. When the data of the other 80 classes is halved or doubled, a probe class's loss changes by −0.19-0.19 and +0.17+0.17, respectively.

Changes in the revised PDF:

  • Section 5.1, Limitations (page 10): the cross-class term and its size.
  • Appendix C.6 (page 31): the full experiment.

Weakness 2 and Question 3. Are the reported intervals conditional on one source fit and one calibration run, and how much do the gains move when the source measurement, the calibration seed, or the calibration allocation is repeated?

Yes, the reported intervals are conditional on one source fit and one calibration run. When both are repeated, the gains remain where the paper reported them: over 20 end-to-end replications per target (5 retrained source pairs ×\times 4 calibration runs) the reduction is 39.1% ±\pm 1.5 and 23.1% ±\pm 1.2 (39.1% and 22.7% in the paper), and PACE wins every held-out run in all 40.

We repeated each measurement you name on the two locked targets of the paper. Entries are RMSE reduction against count-only.

What is repeatedTiny-ImageNet →\to ConvNeXt-TCIFAR-100-LT →\to WRN-28-10
Nothing (paper; interval over held-out runs)39.1% [38.0, 40.3]22.7% [20.5, 25.1]
Calibration seed (same allocation, 3 new seeds)39.4%, 36.2%, 39.2%23.0%, 22.9%, 23.0%
Source runs (5 seeds) and calibration run (original and 3 new seeds), all 20 combinations; mean ±\pm SD [min, max]39.1% ±\pm 1.5 [36.1, 41.0]23.1% ±\pm 1.2 [21.2, 24.4]
Source subset and calibration allocation, resampled from existing runs; median [95% interval]37.8% [34.9, 40.4]23.7% [19.2, 27.2]
Configurations in which PACE outperforms count-only20 of 20 and 364 of 36420 of 20 and 300 of 300
  • The comparison with the pin is unchanged: on WRN-28-10 the pin remains more accurate than PACE in all 20 replications (by 1.8–8.6%), and their average outperforms the pin in all 40.
  • Resampling the source subset and the calibration allocation widens the intervals by a factor of 1.7–2.3.
  • Over all 19 studies (8,494 resampled configurations), PACE outperforms count-only in every configuration in 18. The exception is 20 Newsgroups, the weakest case of the paper: 4.8% [−1.2,12.3][-1.2, 12.3].

Changes in the revised PDF:

  • Section 5, "Robustness to Training Noise" (page 8).
  • Appendix C.3 (pages 28–29, Table 22): the end-to-end replications.
  • Appendix C.4 (page 29, Table 23): the resampled intervals for all 19 studies.

We hope this addresses your concerns, and we would be glad to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comment to all reviewers

We thank the reviewers for their time and their constructive comments. We are very pleased that Reviewer aCHP finds our separation of count effects from persistent class effects, and the question of whether the latter transfer across protocols, "a useful extension of aggregate learning curves" with "a concrete forecasting use". We are also glad that Reviewer EYHU finds the problem "clear and practically relevant" and the prospective experiments "compelling", and that Reviewer hFB5 finds the goal of the paper "crystal clear". We further appreciate that Reviewers aCHP and EYHU acknowledge the transparency of the paper about the boundaries of its claims.

In response to the reviewers' comments, we made the following changes. The new locked targets were designed before training, analyses of existing runs are labeled as retrospective in the paper, and no result of the original submission changed.

  1. More prospective evidence (all reviewers). We added five new locked targets (three in vision and two language models). Together with the two targets of the submission, PACE improves over count-only prediction on all 72 held-out runs of the seven targets.
  2. Repeating the whole procedure (aCHP, EYHU). We retrained the source and calibration runs with new seeds. The gains remain at 39.1% ±\pm 1.5 and 23.1% ±\pm 1.2, against 39.1% and 22.7% in the submission.
  3. Information-matched baselines (aCHP). We compared PACE with one-run raw and count-adjusted source vectors. PACE is more accurate than both on all seven locked targets.
  4. Effect of the other classes' allocation (aCHP, EYHU). A designed experiment with 50 runs shows that it accounts for 2.5% of the variance of class difficulty.
  5. PACE and the target pin (EYHU). A fusion weight chosen without held-out outcomes combines the two, and the fused forecast outperforms the pin in every classification and language-model study.
  6. Tests of the symmetry mechanism (EYHU). We tested Theorem 2 in simulation and with a 126-run experiment on CIFAR-100.
  7. Clarity (hFB5). We rewrote Section 3 with definitions and a worked example before any theorem, redrew Figure 1, and now present Theorems 1 and 3 as elementary identification conditions.

Detailed answers are given in the individual responses. In the revised PDF, changes are colored by reviewer: blue for aCHP, red for EYHU, green for hFB5, and violet for changes shared between reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.