- Rating
- 2(−1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks how to allocate resources to a prediction head attached to a shared representation: how much persistent state to keep ( FP32 coordinates), how often to read the shared features (a fixed-width input packet every steps). It first gives a theoretical split of the cost into two parts. For a decoder whose output between reads is an affine function of the persistent state, the lower bound on the gap risk is . The first term is new information that arrives only after the last read, which can be reduced only by reading earlier. The second term is the predictable spectral tail cut off by the finite state, which can be reduced by adding state. On top of this, the paper proposes a concrete method: train a multi-step predictor of the task's next 16 answers, project its predictions onto a -dimensional subspace to obtain a deployable head for each candidate , estimate each candidate's risk upper bound using 65,536 independent calibration episodes and an empirical Bernstein inequality, and select the cheapest combination satisfying . This removes the need to train a receiver for every candidate.
On validation, main results come from 12 new synthetic environments. The compiled method's cold preparation is 20.29× faster than a full receiver grid, the selected resource cost is 1.038 times that of the grid, and all 432 price queries pass the independent test; against a cost-ordered early-stop search, the speedup is 12.11×. Interventions on 24 worlds show that adding state helps under low innovation and shortening the read interval helps under high innovation, matching the theory. In comparisons, the method that compresses future answers is 32–40% cheaper than the methods that compress current features (P/H), but it essentially ties with the stronger per-cell shared supervised grid. The authors also honestly report unfavorable results: when the calibration size drops to 16,384, 216/432 queries find no certified cell; independently trained receivers fail on high-harmonic tasks; and on frozen DAVIS video features, simple sample-and-hold achieves a lower MSE (by 0.0052) at no greater cost.
Critical strengths
Strength 1: Paper separates state capacity and read frequency into two bottlenecks. Proposition 1 states that, for a decoder whose output between reads is an affine function of the persistent state, the lower bound on the gap risk is . The first term is new information that appears after the last read, which increasing the state can't recover. The second term is the predictable spectral tail cut off by the finite state, which increasing the state can reduce. In the low-innovation world, increasing from 1 to 7 reduces MSE by about 0.167; in the high-innovation world, shortening from 8 to 2 reduces MSE by about 0.181, while can't get past the information floor. These results come from 24 independent worlds, and the statistical unit is the world.
Strength 2: I was able to verify the key parts through the code the authors provided. I spent some time understanding the whole code, on two tasks I recomputed the projections (UUᵀ differing by about ), the calibration upper bounds for all 24 cells (differing by about ), and the risk on 524,288 independent test episodes. I think the results show that the selection rule plus the empirical Bernstein certification is reproducible.
Critical weaknesses
Weakness 1: Paper's biggest speedup is measured against a weak baseline. All of the evidence that the method works comes from a synthetic environment. The target is a combination of a finite number of circular modes, and the output between reads happens to fall within the affine class characterized by Proposition 1. Authors themselves also acknowledge some results fail on high-harmonic tasks. The only real-data experiment is on DAVIS video features, its result is unfavorable to the method: simple sample-and-hold has an MSE 0.0052 lower than F at no greater cost, the target was adjusted only after the development videos had been inspected, and the 30 videos cannot certify the population risk. 20.29× is measured against a full grid in which the receivers of all 19 cells are trained, which is a very easy baseline to beat. A more reasonable comparison is early-stop search, for which the speedup falls to 12.11×, and that comparison was in turn specified after the fact on old worlds whose results had already been seen; the more conservative F-then-A path achieves only about 5.6×. In addition, this advantage requires 65,536 calibration episodes, and when this is reduced to 16,384, 216/432 queries find no certified cell.
Weakness 2: The lower bound in Proposition 1 holds only for the ideal affine class. Authors themselves point out that the F, P and H actually executed all carry a final output clip and belong to , which is not in this class. Paper also reports that a correctly clipped nonlinear receiver crosses the affine ideal minimum on seven tasks, showing that this bound is not a restriction on all legal receivers. The budgets selected by the method are certified only by an upper bound and not by a lower bound: at , the blind lower-bound rule excludes all cheaper cells in only 8 of the 72 tasks in each family, so the high agreement with the independently trained grid is mainly an empirical phenomenon. At the same time, the advantage the paper shows is also quite limited: F is cheaper than P/H, but these two comparators merely apply principal-component compression to the current features and have no task-supervised encoder; and against a stronger comparator (a shared grid trained separately for each cell, S-per-cell), F only ties on 70–71/72 tasks.
Questions for authors
-
In the only real-data experiment (DAVIS), sample-and-hold achieves a lower MSE at no greater cost. On what kinds of real tasks do you expect compressing future answers to outperform simply holding the current estimate, do you have any real-data results showing this?
-
What are the side-by-side timings of F, the full grid and early-stop search on the same machine? Also, in the stored results, F-then-A has test results identical to Direct-A; did F-then-A actually retrain an independent receiver?
-
Proposition 1 holds for the ideal affine class, while the executed F, P and H carry an output clip, and selection relies only on upper-bound certification. What role does the theory play in the method's selection rule, beyond motivation?
2 comments
Thank you for your detailed rebuttal and for providing the additional experimental results. I have read your response and examined the newly added experiments. However, I still have reservations regarding the reliability of the experimental data in your paper. I am maintaining my current rating.
Thank you for checking the projections, 24 calibration bounds and test risks on two tasks. We added a same-packet temperature comparison and the requested same-machine timing study, and clarified how the bounds guide budget selection.
Forecasting and hold on real data. We added a same-allocation comparison on a previously examined Jena station. Forecast compression (F) with one value and eight-hour reads gives normalized MSE .0507 versus .1451 for holding the latest temperature. At sixteen hours, F improves from .1186 with one value to .0694 with two, meeting the .10 target. A supervised same-packet receiver also shows this exchange. The 17,462 temperatures include read steps, packets contain source-side history, and errors use training-temperature variance (revised §4.2 and Appendix H). In the original DAVIS comparison on 30 held-out videos, hold gave .00519 lower MSE at no greater cost (original §4.5 and revised §4.7).
Same-machine timing. We added a side-by-side comparison in one process on one logical CPU (revised §4.3 and Appendix E). On 24 preselected tasks from 12 previously examined worlds, F averaged 1.439 s/task, the full grid 29.117, and early-stop search 18.902. Timing includes fitting and calibration and excludes the shared setup and test.
Refitting and calibration. We now explain in Appendix E why F-then-A has the same test result as Direct-A at a shared cell. It refits the receiver using Direct-A’s training data, seed and deterministic recipe, then calibrates it on an independent lane. In the original comparison, F and Direct-A each certified all 432 choices with 65,536 independent calibration episodes. At 16,384 episodes, F certified 216/432 and Direct-A 380/432 (original and revised Appendix E).
Theory and budget selection. We clarified the role of each bound in revised §§2–3. The selector chooses the cheapest candidate with a passing calibrated risk upper bound. The affine lower bound rules out a budget when it exceeds the target for current-packet writers whose complete between-read output lies in a fixed affine space of dimension at most the state size. At .04, it excludes two values at sixteen steps in the original example, under its source and numerical conditions. It motivates projecting forecasts onto their leading directions. At read price 4, the original blind lower bound excludes every cheaper cell in 8/72 tasks per source family. The upper-bound rule gives feasible choices within that same evaluated grid (original and revised Appendix F). The main compiler’s gap outputs are unclipped (original and revised §3). F/P/H comparison heads use a final output clip (original §4.3, revised §4.4 and Appendix F in both).
Supervised-grid comparison. In the original comparison, after 38,000 updates, the supervised grid ties F on 70–71 of 72 tasks per comparison and wins the rest (original §4.3, revised §4.4 and Appendix F in both).