- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies teacher–student state mismatch in recurrent ternary transformers and proposes Megrez to reconstruct teacher targets before calibration. The key technical idea is a neat formulation of target reconstruction as a constrained orthogonal Procrustes problem with a closed-form solution in a low-dimensional subspace. Experiments on several recurrent ternary models show substantial target alignment improvements and promising quantization results.
Critical strengths
-
The method is well tailored to the shared-weight setting of recurrent ternary transformers, and the idea of adapting the intermediate targets to the current student is intuitive.
-
The authors formulate target reconstruction as a constrained orthogonal Procrustes problem in the relevant low-dimensional subspace, resulting in a neat closed-form solution.
-
The experiments provide evidence at both the geometric and end-to-end levels, showing improved target alignment and promising quantization performance. Weaknesses
Critical weaknesses
-
The paper shows that intermediate teacher and student targets can be mismatched, but it is less clear whether this mismatch is actually a major optimization bottleneck for ternary calibration. More direct evidence connecting target mismatch to calibration difficulty would make the motivation much stronger.
-
The concept of “readout-preserving” could be stated more precisely. The proposed transformation preserves the constraints imposed by the native readout, but it does not mean that the reconstructed targets are fully equivalent to the original teacher targets. Making this distinction clearer would help better convey what is actually preserved by the reconstruction.
-
The connection between geometric alignment and quantization gains is still somewhat indirect. The substantial reduction in target reconstruction error does not consistently translate into better PPL or accuracy, as shown by the matched controls in Table 2. This makes it difficult to tell how much the geometric improvement itself contributes to the final quantization performance.
Questions for authors
-
Is Megrez a general solution for recurrent quantization, or does the constraint need to be designed separately for each readout structure?
-
How much of the final gain comes from target reconstruction itself, rather than the auxiliary loss or target refresh strategy?
2 comments
All four methods match the teacher's original output probabilities using Kullback–Leibler (KL) divergence. KL-only uses only the output loss. Fixed adds the original intermediate targets. Free Alignment (Free) and Megrez use reconstructed targets, with Megrez requiring the teacher's frozen readout to return its original output on those targets.
W3 / Q2. How much of the prediction gain comes from reconstruction? The available controls estimate an increment for each calibration setting. Megrez versus Fixed measures switching from original to periodically reconstructed targets while retaining intermediate supervision. Megrez versus KL-only measures adding the full intermediate supervision, and Megrez versus Free measures the readout constraint at the same refresh rate. The Fixed comparison changes target construction and refresh together. Isolating refresh would require varying its schedule within one target family.
Matched training evidence. Below are both LoopFormer cohorts on 1,024 held-out Colossal Clean Crawled Corpus (C4) documents per cohort after 256 matched updates. Each cell gives mean Megrez-minus-control negative log-likelihood (NLL) in nats/token, with wins out of three training blocks.
| Cohort | Inference calls | vs KL-only | vs Fixed | vs Free |
|---|---|---|---|---|
| Existing checkpoints | 4 | −0.239 (2/3) | −0.305 (3/3) | −0.441 (3/3) |
| Existing checkpoints | 8 | −0.040 (1/3) | −0.156 (2/3) | −0.092 (3/3) |
| New training blocks | 4 | +0.415 (1/3) | +0.203 (1/3) | −0.168 (1/3) |
| New training blocks | 8 | −0.091 (2/3) | −0.217 (2/3) | −0.183 (2/3) |
The eight-call mean against Fixed favored Megrez in both cohorts. At four calls the sign changed. In the new four-call cohort, adding fixed intermediate supervision increased mean NLL by 0.212 in the table's units (Fixed minus KL-only). The increments vary by cohort and depth, so a single fraction of the total calibration gain would be misleading (Table 2 and Appendix H.3).
Ouro one-hop color tracking. At four calls, Megrez lowered one-hop candidate NLL relative to Fixed in both earlier 256-update blocks, by 0.09107 nats/question on average. The rate of answering both counterfactual questions correctly rose by 5.08 percentage points. We evaluated both new training blocks on a separate set of 256 graph pairs. Megrez had lower NLL than all three controls in one block and higher NLL than all three in the other. The second result persisted after retraining on 4090D GPUs. Appendix H.4 gives the earlier mean and the differences for each new block.
What changes when we hold the evaluation data fixed? We trained Ouro for the first 32 updates of the original 256-update schedule. At initial auxiliary weight , we evaluated the original and newly trained blocks on the same scoring device. The table reports differences in nats/token.
| Training block | Windows | vs KL-only | vs Fixed | vs Free |
|---|---|---|---|---|
| Development | 48 development | −0.05507 | −0.00480 | −0.03624 |
| Development | 128 held-out | −0.03321 | −0.00206 | −0.02435 |
| New 1 | 48 development | +0.00010 | +0.00138 | +0.00339 |
| New 1 | 128 held-out | +0.00021 | +0.00086 | +0.00383 |
| New 2 | 48 development | +0.00789 | −0.00231 | +0.00410 |
| New 2 | 128 held-out | +0.03719 | −0.00587 | +0.00679 |
All nine contrasts kept their sign across evaluation sets. Rankings changed across training blocks on the same evaluation set. The roles of training-window selection, order, and numerical variation remain unresolved (Appendix H.6, p. 27).
Thank you for asking how geometric alignment connects to calibration and which properties a reconstructed target retains.
What the four labels mean. All four methods match the teacher's original output probabilities via Kullback–Leibler (KL) divergence.
Free and Megrez both use one orthogonal map per token to bring the teacher's recurrent states closer to the current student's. The student fits the resulting targets through its shared ternary weights.
W1. Do reconstructed states help the student make predictions? We replaced intermediate states at all token positions in frozen KL-only students, after call two in Ouro and after call four in LoopFormer, then ran each student's remaining calls. Each comparison uses 32 windows per corpus and measures full-vocabulary next-token negative log-likelihood (NLL).
Values are in nats/token. With sparse substitution in Ouro, the advantage over Free was 0.0212 on C4 and 0.0151 on WikiText-2. The Megrez–Fixed interval spans zero on Ouro C4. The LoopFormer student scored best with Fixed, even though Megrez lowered the target mismatch by 53.4% on C4 and 55.9% on WikiText-2. Direct substitution tests what frozen students can do when given these states. Learning to produce them requires changing shared weights, so whether mismatch is a dominant training bottleneck remains open. See Appendix H.4 (p. 26) for the Ouro intervention and the text after Eq. (7) (p. 5) for the distinction between the fixed-state solve and the shared-weight update.
We also measured auxiliary gradients from different recurrent calls at one common Ouro initialization. Each of eight windows supplies six call pairs.
The constraint changes gradient agreement in this diagnostic. The result leaves its effect on the final training outcome open.
W2 / Q1. What is preserved, and which readouts are covered? In exact arithmetic, the frozen teacher's native readout returns its original output when applied to the reconstructed targets. One orthogonal map per token preserves the inner products among its normalized teacher states. The individual targets can move, and the recurrent model may be unable to generate the resulting trajectory. The explanation following Proposition 1 (pp. 4–5) states this scope.
The solver is shared once the fixed directions have been derived for each readout. In Ouro, Megrez fixes the gain-adjusted gate direction and original weighted aggregate in normalized coordinates. This preserves the gate weights and the input to the output head. In Parcae and LoopFormer, fixing the final-state sequence preserves the downstream readout, including the input to Parcae's layers after the recurrence.
The first two rows show Ouro's initial snapshot from Figure 4 (p. 8), where relative logit drift was measured in half precision. The last row is a separate frozen-snapshot diagnostic over 64 held-out windows.