- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper identifies a representation bias in model-zoo aggregation: duplicating checkpoints or changing the sampling density and parameterization of model families can alter ensemble predictions without meaningfully changing the underlying predictive family. The authors propose Representation-Invariant Measure Ensembling (RIME), which combines exact predictive quotienting with Hellinger arc-length weighting to construct a representation-invariant base measure. Cross-fitted Brier-loss evidence is then used to adapt the ensemble weights. The paper establishes theoretical results for exact clone-splitting and monotone reparameterization invariance, along with approximation bounds for finite-grid refinement. Experiments on PACS, NICO++, Office-Home, and ACSIncome demonstrate strong stability under registry transformations.
Critical strengths
-
The paper identifies a subtle but meaningful limitation of conventional ensemble aggregation, where arbitrary registry choices can influence predictions. The proposed combination of predictive quotienting and Hellinger arc-length weighting provides a mathematically motivated solution, supported by explicit invariance guarantees.
-
The experiments directly evaluate the proposed invariance properties through checkpoint duplication, near-cloning, and grid refinement. The results demonstrate substantial improvements in representation stability without meaningful degradation in predictive performance. I also appreciate the careful experimental analysis and transparent discussion of the limited predictive gains.
Critical weaknesses
-
The invariance guarantees are conditional on a fixed family partition, path ordering, predictive embedding, and target-input slice. Changing these specifications can alter the resulting aggregation. While the authors acknowledge these limitations, they restrict the broader applicability of the representation-invariance claim.
-
The strongest improvements arise from deliberately constructed registry perturbations, whereas predictive gains over the matched family-balanced baseline are very small and not robust under conservative target-level clustering. Additional evidence from naturally occurring model registries would help establish the practical significance of the proposed method.
Questions for authors
-
How sensitive is RIME to different reasonable family partitions or path definitions? Can the authors provide guidance for specifying these structures in practical model zoos where the underlying families may not be explicitly known?
-
How frequently do the identified registry biases meaningfully affect predictions in naturally constructed model zoos, without deliberately introducing clones or refining selected paths?
-
What motivates Hellinger geometry over alternative invariant predictive embeddings beyond the stated analytical properties? Could the authors clarify when different geometric choices might lead to substantially different ensemble predictions?
4 comments
Response to Reviewer TTuJ (2/2)
Q2: "How frequently do the identified registry biases meaningfully affect predictions in naturally constructed model zoos...?" Frequently at the prediction level, but rarely at the decision level. E2 (App. Z.6) audits unedited registries: 1080 PACS cells, plus ACSIncome, Office-Home and NICO++ (the latter with 2 of 5 bank seeds).
- Natural exact collisions occur in every cell (PACS: 41 rows → 34–40 atoms).
- Predictions (compared with RIME on PACS, 4 / 16 labels/class):
- Per-cell TV exceeds 0.01 in 58% / 17% of cells for FABLE and 80% / 19% for flat rows (median TV for FABLE 0.011 / 0.0058).
- The predicted class changes on about 1% / 0.4% of test examples.
- Performance: mean ΔBrier stays within ±0.00078.
- Decisions: releases differ in ≤ 4/540 cells per budget. In every such case RIME releases and the row rule abstains.
- External registries: |mean ΔBrier| ≤ 0.00017, with no release changes (all methods abstain there).
- Mechanism. FABLE and RIME differ only inside path families, where arc mass gives little weight to near-converged low-penalty endpoints. Flat rows instead change whole-family weights by row count: on PACS, a 6-row path family gets 1.6× RIME's family mass and a singleton family 0.28×.
E5 (App. Z.7) pools five NICO++ training seeds into a large real zoo of 101–111 checkpoints per registry. All base measures agree within 0.001 Brier on every architecture–target unit, because the evidence puts ≥ 0.98 posterior mass on the fallback. Yet the flat prior already carries a 1.6× domain imbalance (0.21 vs 0.13) from uneven checkpoint saving; FABLE and RIME give 0.18 per domain. The bias exists, but in these zoos it is swamped by evidence. It becomes large when the registry is edited (W2).
Q3: "What motivates Hellinger geometry over alternative invariant predictive embeddings...? ...when different geometric choices might lead to substantially different ensemble predictions?" Motivation beyond analytical convenience (App. Z.1). With the target slice fixed, is the Fisher metric of the joint law over (slice input, class). Two facts follow:
- Uniqueness of the metric. By Chentsov's theorem and its conditional version (Montúfar, Rauh & Ay, 2014), this metric is the unique Riemannian metric, up to scale, that is invariant under homogeneous conditional embeddings. Examples: splitting a class into fixed-proportion sub-classes, or replicating the slice. The Hellinger chord is exactly invariant under these operations; probability-Euclidean and centered-log chords are not.
- Jeffreys prior. The arc measure is the Jeffreys prior of the path (Remark after Thm 1).
Scope of this claim. The theorem fixes the metric, not the finite chord. All -divergences share the same local Fisher form, which is why root Jensen–Shannon stays closest to Hellinger (below). The theorem also says nothing about which geometry gives the lowest loss.
When geometries differ (E6, App. Z.8). The path speed weights the motion of class by (Hellinger), (probability-Euclidean) or (centered-log), so geometries can disagree only on segments that move probabilities near the simplex boundary (likelihood ratio ≤ if ). We measured this on all 720 native PACS cells (6 seeds, 4 and 16 labels/class), with criteria fixed in advance:
- Root-JS never matters (mass provably within 1.2× of Hellinger; max TV 0.0053). Euclidean barely matters (mean TV 0.0013, max 0.070; 1 of 720 cells substantially different).
- Centered-log (with the paper's clipping) changes individual cells: mean TV 0.0079, max 0.16, predicted class changes on up to 4.1% of a cell's test examples. 37% / 17% of cells are substantially different (mean TV > 0.01, a release flip, > 1% argmax change, or |ΔBrier| > 0.005).
- Where: it moves base mass from the strongly regularized start of the logistic, ridge and entropy paths (0.4–0.6× Hellinger) to their weakly regularized, near-boundary ends (C 10→100, λ 0.01→0, η 4→8: 1.6–2.5×).
- Evidence gates it: per cell, centered-log's predictive TV is uncorrelated with its base-measure TV but tracks posterior mass on the reweighted families (Spearman 0.75 / 0.55).
- Decisions: mean ΔBrier vs fallback differs from Hellinger's by < 0.001 in every seed and budget. Releases flip in 4 of 2160 comparisons, all knife-edge (Hellinger's log e-value 5.08–5.31 vs threshold 5.075).
We would be glad to answer any further questions.
Thank you for the clarifications and additional experiments, which substantially address my questions. While practical gains beyond representation stability remain limited, the rebuttal strengthens my assessment. I will maintain my rating of weak acceptance.
Thank you very much for taking the time to review our clarifications and additional experiments! We truly appreciate your constructive feedback throughout the process and are glad that our responses strengthened your positive assessment of our work.
Response to Reviewer TTuJ (1/2)
Thank you for the careful review. We address each point below; all new results are in App. Z of the revision (marked in blue).
W1: "The invariance guarantees are conditional on a fixed family partition, path ordering, predictive embedding, and target-input slice..." Agreed: the guarantees are conditional, and the paper should say more precisely which guarantee depends on which declaration.
W2: "...predictive gains over the matched family-balanced baseline are very small and not robust under conservative target-level clustering. Additional evidence from naturally occurring model registries would help..." Agreed on the gains. Against FABLE, RIME ties at 4 labels/class and gains 0.00069 Brier at 16 labels/class, which is 0.7% of the adaptation gain. The interval crosses 0 under target clustering (Table 3). Our claim is "free, not profitable": RIME is insurance against registry bookkeeping. We added evidence from natural registries (E2, E5; details in Q2) and from registry edits with real refits:
Q1: "How sensitive is RIME to different reasonable family partitions or path definitions? Can the authors provide guidance...?" Sensitivity (E1, App. Z.2). We applied 10 declarations to the same fitted PACS registry: merged singletons, merged logistic-head paths, paths split in half, no paths, two label-free automatic declarations, and three random path orders as a control.
Guidance (a new paragraph in the main text, details in App. Z.2):