- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
COMPASS makes communication cheaper for distribution post-training through 3 components:
-
it compresses the dimensions of the activations of fast circuit at pipeline boundaries (alternating ones in SFT) by projecting onto a low-rank basis that the anchor estimates from its uncompressed activations via power iteration
-
the anchor maintains a running average of its gradients and sends the signs of that average (directional information) to the fast circuit. These signs are then used correct the fast circuit gradients, flipping the direction of its gradients if the anchor and fast circuit gradient directions don't align (\alpha < 0.5)
-
For SFT, only top-10% blocks of weight change between the fast circuit weights and the anchor weights, quantized to int8, are used to update the anchor's weights, with error feedback. For RLVR, COMPASS additionally does anchor weight forecasting, which linearly extrapolates weights from the two most recent available snapshots and uses them to compute the basis (1) and signs (2) with an aim to reduce their staleness
Critical strengths
-
full-parameter distributed post-training of existing open-weight models without architectural changes is a timely and under-studied problem
-
the proposed learned activation basis is shown empirically to outperform random mask used in prior work at the same compression ratio
-
the anchor sign correction yield large gains in accuracy for RL
Critical weaknesses
my main concerns are:
-
Lines 784-792, the authors mentioned that their main result in Table 2 are "synthetic draft placeholders, with matched measurements still pending". In other words, apart from NarrativeQA, code, and SQL metrics, the rest of the numbers in the table appear to be placeholders. Furthermore, in the same paragraph, the authors wrote that "earlier unspecified Code result was 0.475 for Dense, 0.510 for the previous method, and 0.497 for COMPASS" but this matches nothing in Table 2.
-
Line 712 and Line 725, the authors claim to run their experiments with multiple seeds and report mean and standard deviation but their results only report a single number. Also, Appendix A is written using future tense, e.g. "will use", "will differ", "will be disabled" making the reader confused what has been run.
-
Table 9 shows SQL increasing from 0.67 to 0.79 when K drops from 80 to 20 while Table 2 reports 0.79 at K=165. Furthermore, Appendix F timing model gives K~=132.7 and describes K=35 as unverified (Line 919). Which K was actually used?
-
The 128x applies only per compressed boundary but only alternating boundaries are compressed in SFT (Lines 772-773), does the abstract's claim of "more than two orders of magnitude" still hold for end-to-end reduction or is it overclaimed?
-
Table 10 Qwen3-4B-Base shows that COMPASS performance is below Dense in all cases but is written as "on par" in the conclusion
Given these inconsistencies between the claims and reported results, and the integrity and reliability of the main results, I recommend rejection. I am also open to raising my score if the authors can provide complete mean and standard deviation results as described in the Appendix, clarify the anchor delay used in the main experiments, and release anonymized code so I can independently verify their numbers.
Questions for authors
See weaknesses
Thank you for the detailed review. The inconsistencies you list are fair.
The Appendix C sentence was written while we were drafting the paper. The results in the table are derived from our actual empirical results. We have now included an anonymised code in the supplementary materials for reproducibility. In the future version we will revise the manuscript carefully. This was an honest mistake on our part, and we apologise for it,
All our training runs used a single seed, so we cannot provide means and standard deviations.
The supervised runs used an anchor delay of K = 20 fast steps, and Table 9 varies it from 20 to 80. The Appendix F values are estimates from a simulated timing model, not measurements.
The abstract's figure refers to payloads at compressed boundaries. Only alternating boundaries are compressed, so it does not hold end to end.
On scale, we agree that "on par" does not hold at the larger scale. At 1.5B, after 200 updates, COMPASS and Dense have similar ten-benchmark means (30.03 and 29.36), and the differences go both ways across benchmarks, so we think "on par" is fair at that scale, from one run each. At 4B, after 100 updates, COMPASS is above the base model on the mean but below Dense on every benchmark (34.05 against 38.64). We tuned the settings at 1.5B and used them at 4B without retuning. The conclusion should have reported each scale separately.
In future we will come with a strong submission addressing all the reviewer concerns.