- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
As I understand it, the paper studies a setting in which the question is when to write data into a model's weights and which data to combine for such write operations. Different ways to add data to the weights are considered: train on everything from scratch, train only on a batch of the new data, or train the new data together with a few examples of the previous data. Full retrain is the optimal baseline. The paper shows that training only on a batch of the new data fails, but that training together with a few past examples gives a good overall tradeoff.
Critical strengths
The general topic of this paper is interesting even though the precise technical setting is much more narrow.
Critical weaknesses
I struggled a lot with this paper. And maybe that is partly my fault, but I found the paper incredibly difficult to read in a way that is not easy to articulate. Crucial definitions that would belong in the introduction as formally describing the setting, are only found in the appendix. The tone of voice in large sections of the paper feels robot like and it is almost like the text wasn't designed to be a self-contained paper with a nice exposition for a human reader. "Appendices A and B build the world (economics, notation, substrate); C through F prove the economics of frames and schedules (ledger, gauge split, dominance, scheduling)": "build the world"? "substrate"? All over the paper, there is a lot of economic terminology in place of mathematical one ("ledger", "tax", "premium", "currency").
Appendix P really needs to be reformatted.
I wished I could say something more substantive about the content of the work, but, sadly, I am afraid I was unable to get that far despite spending a considerable amount of time on reading the work.
Questions for authors
I did not fully understand the cost measure for W2. Do you not need increasingly many examples in the anchor as the number of classes grow? And doesn't this increase the cost of the anchor over time?
the paper answers Questions 1 and 2 and half of Question 3. the population ledger is exact, the middle is measured and negative, and the separation depends on the charge.
Dear Reviewer M3JH,
You asked three questions and raised three concerns. The paper answers Questions 1 and 2 in full, and half of Question 3. Your first two concerns are fair, and the paper acknowledges both; the third is right too, and we name that gap openly.
Question 1: the residual excess error. At the population level the answer is exact, not a bound. The Section 4 ledger prices every cross-batch pair in closed form, and Proposition 13 splits that price in two: a per-class shift, which a single offset removes completely, and a misalignment of direction, which no constant can fix, because constants move thresholds, not directions.
The residual is already inside the competitive analysis: its floor is integral to Theorem 4's separation. Under every composition and correction the flooded pair's residual stays above that floor, and Proposition 33 caps it at 0.5807 (Appendix P.2). This exactness holds in the ideal case, but it stops on finite data. The proof of the anchored escape leaves anchor noise outside its scope by design, and Section 10 says plainly that the ledger's level does not carry over to learned features. A distribution-free finite-sample bound is still open problem; we do not know one yet.
Your second concern and Question 2: how expensive replay really is. You are right, and the paper says so. The scheduling guarantee treats setup as a fixed price, but a covering anchor must store examples of every class written so far, so its cost grows with the store. The registered cost model charges a write by the classes it serves, which makes full coverage grow with the square of the store-to-batch ratio (E-L2-11); the experiments therefore ran both arms (E-L2-12).
The measurements agree with the model. A covering anchor's setup price came in at 147.1 times the plain write's (152.2 and 155.8 on other seeds), growing with the store as predicted. On 15 adversarial streams built to make undoing as costly as possible, the covering policy never once beat the break-even rule; its cost ratio, 47.19, stayed within the 156.84 the theory allows. Section 9 reaches the same conclusion: at prices that high, the promised improvement has no practical use.
Nor does partial coverage rescue it. At the two extremes the theory is settled: an anchor covering every class restores the classical guarantees at no extra cost (Proposition 5), and writing with no anchor at all has no bounded ratio (Theorem 4). Between them there is no theorem, only measurement, and the measurement is negative.
Before any run, E-L2-11 registered the prediction that below break-even coverage a covering write leaves pairs worse off than no write at all, and that is what happened: at coverage depth 64 the covering write left 0.0020 excess over the joint refit, while the unanchored baseline sat at 0.00026, indistinguishable from zero at a hundred classes (t = 0.56). Break-even coverage measured 128 per class, 1.28 times the store; the grid never reached that high, so 128 is a floor. Where coverage applies, it does beat size: a one-exemplar covering anchor left 0.0079 where an equal-sized non-covering anchor left 0.0440 (paired z = -7.7 over 75 rows). We also tried ranking which classes to cover; the rule performed at chance (Spearman 0.0058) and lost its registered bars. The middle is unresolved, and the submission says so directly.
Your first concern and Question 3: what the separation depends on. It depends on what a query is charged, and the submission states this beside the theorem. Theorem 4 charges the balanced-prior bill, and under it the separation holds. Under a one-sided charge the flood is free: a policy can steer the batch composition until the balanced error reaches 1/2 while every flooded-class query stays correct, and the ratio is bounded (Remark 9).
On the operator side, the separation binds the unanchored writer; anchoring and refitting are the escapes, and the paper prices both. For a general query weighting we have a sketch but no theorem, and we will not promise one in this window; for multiclass loss we have nothing yet. The experiments do carry the unarranged flood, in which classes arrive with heavy-tailed popularity and are queried in proportion: no policy exceeded its proved bound on those 15 streams, and on CIFAR-100 class means the unanchored ratio grows to 775.2 against the anchored 2.0.
Your third concern is correct. The paper prices the pieces separately: the anchored arm reduces to classical scheduling (Proposition 5), the mixed-operator bounds price mixing (Theorem 31 and Corollary 32), and the ledger prices which items belong together. A single policy that decides when to write, which items to combine, and which operator to use is an open problem.
Thank you very much for working through the merge theorem and Example 2, and for crediting the failure reports. If these answers change your view of the paper and your assessment, we would be very grateful.