- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MV-HC initializes separate KromHC residual streams from BPE embeddings and causally aligned auxiliary-token embeddings, then combines the final streams through a learned gate. It retains the original residual mixer and shared Transformer branch. Experiments from 60M to 1.5B parameters report improved validation and downstream performance, with component, tokenizer-choice, and stream-similarity analyses and approximately 1% throughput overhead relative to KromHC.
Critical strengths
S1. Simple and compatible construction. The main tokenizer retains the prediction interface, auxiliary alignment explicitly respects the observed prefix, and each layer executes the expensive Transformer branch only once.
S2. Reported improvements across model scales. MV-HC improves validation BPB at 60M, 186M, and 1.5B. At 1.5B, BPB decreases from 0.628 to 0.617, while commonsense and broader evaluation averages improve from 55.3 to 56.7 and 35.1 to 36.7, respectively.
S3. Useful existing controls. Table 7 includes parameter matching, gating, and auxiliary-fusion baselines. Tokenizer comparisons and late-layer stream similarities provide additional diagnostics beyond aggregate benchmark results.
S4. Small incremental cost over KromHC. At 1.5B, the reported additional parameters increase from 1.36M to 2.05M, while throughput decreases from 63.5k to 62.9k tokens/s. This makes the extension inexpensive within the existing HC architecture.
Critical weaknesses
W1. Limited methodological novelty. The residual mixer and layer update are inherited from KromHC; the changes are auxiliary-embedding initialization and final gating. Combining character/subword representations also has substantial precedent, including the cited CharBERT. The described lookup/projection/pooling path offers a compositional token representation, leaving the conceptual advance relatively modest.
W2. The decisive routing control is missing. Table 7's KromHC+AuxFuse reaches 0.699 BPB/48.2 CORE versus 0.696/48.6 for full MV-HC, but lacks the latter's final gate. An AuxFuse+gate control is needed to attribute this remaining gain to keeping views in separate streams. Lower stream cosine similarity alone does not establish useful information diversity.
W3. Baseline provenance and CORE reporting need reconciliation. The small-model baseline rows in Tables 2–3 largely match the published KromHC tables, despite the statement that all baselines were trained from scratch. Meanwhile, Table 1 reports KromHC CORE of 42.4/47.7, versus 9.018/16.872 in the cited paper. This may reflect different runs or metric definitions, but their relationship must be documented. See KromHC, Tables 3–5.
W4. Essential reproduction details are missing. The paper omits auxiliary embedding dimensions, concrete tokenizer configurations, the selected pooling rule, and optimization settings for the new parameters and 1.5B model. ParamMatch and the “shared tokenizer” variant are insufficiently defined; describing tokenizer sharing “across layers” is also unclear when auxiliary tokenization occurs only at initialization.
W5. Scaling beyond small dense models is unverified. Experiments cover 60M–1.5B dense models with four residual streams. Do the gains and approximately 1% incremental throughput overhead persist at larger scales, particularly a 30B MoE? Evidence or analysis of the scaling trend, compatibility with expert routing, and training/inference memory costs would strengthen the practical significance.
Questions for authors
-
Which baseline results were rerun versus taken from KromHC? Please reconcile the differing CORE values and specify the exact metric computation and evaluation protocol.
-
How does KromHC+AuxFuse perform with the same final gate as MV-HC, and is the remaining difference stable across training seeds?
2 comments
Thanks for the response from the authors. I am still concerned with the novelty thus I keep my rating.
Summary. We thank the reviewer. Both questions are answered with new results: AuxFuse + gate (W2, Q2) is run with 3 seeds, and we revised our claims; baseline provenance and CORE (W3, Q1) are reconciled below. We commit to releasing reproduction details (W4), add a scaling analysis (W5), and address novelty (W1). Some points overlap with Reviewer bj7q; we refer to that reply to save space. We hope these responses address your main concerns and encourage you to raise your score.
W2/Q2 (AuxFuse + gate). This was the decisive control. Over 3 seeds at 186M (BPB, mean ± std):
AuxFuse+gate beats KromHC in every seed and recovers ~80% of MV-HC's gain. MV-HC is lower in 2 of 3 seeds, but the remaining 0.0016 BPB is within seed noise (seed-1 CS Avg.: 49.0 vs. 49.3). The gain therefore comes mainly from the causally aligned auxiliary view, not from keeping it in separate streams; we revised Sec. 3.6, App. A.8 and the Conclusion accordingly. We agree that lower cosine similarity alone does not show useful diversity. The controls in our reply to 9Fpt (Q1) test this: noise-induced diversity gives no gain, an unrelated byte view is down-weighted by the gate, and corrupting the aligned view removes the gain.
W3/Q1 (provenance, CORE). All models were trained by us, and the training loss and BPB in Table 1 come from our runs. However, the 60M/186M baseline rows of Tables 2–3 were copied from the KromHC paper instead of being evaluated on our checkpoints. We re-evaluated all eight rows with the same pipeline as MV-HC (table in our reply to bj7q, M1); MV-HC remains best on average at every scale. Our "CORE" column was the unweighted mean accuracy over the 11 commonsense tasks of Table 2, not DCLM's centered CORE, so it is not comparable to 9.018/16.872. It is now named CS Avg. and defined in Sec. 3.3. ΔParams was also recounted; our KromHC overhead now equals KromHC's exactly (240K/959K).
W4 (reproduction). We will add a full configuration table to the appendix (tokenizers, auxiliary dimensions, pooling, gate, optimization, the 1.5B setup, and the ParamMatch and shared-tokenizer variants) and release the code.
W5 (scaling). We have not trained beyond 1.5B or MoE models, and the Limitations now state this. The analysis is favorable. With the same n, MV-HC's per-layer computation and residual-stream memory are identical to KromHC's; MV-HC only changes what fills the streams. Its extra cost sits at the input (byte lookup, d_a×d projection, pooling) and the output (gate) and does not grow with depth, so its relative overhead shrinks as models grow. The parameter overhead beyond KromHC is 192(256+d)+d+1, i.e. 0.20M at d = 768. Extra activation memory is B×S_aux×(192+d), at the input only. MV-HC does not change the attention/FFN branch, so expert routing receives the same aggregated input as in KromHC.
W1 (novelty). We agree the change to KromHC is small; see our reply to bj7q (M4). Unlike CharBERT, which fuses character and subword channels in a bidirectional encoder, MV-HC targets autoregressive LMs: its causal alignment admits only auxiliary units completed by the current position, and the views enter as hyper-connection streams rather than through a fusion module. Our analyses also show which views help (causal coverage, App. A.13) and that diversity alone does not.