- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MCC-OPD for memory-grounded question answering. It contrasts teacher predictions with and without supporting evidence, then uses a mirror KL budget to control the correction strength. The student learns from the calibrated targets through reverse-KL distillation. The paper proves that the budget can be enforced by scalar bisection and evaluates the method on four benchmarks after training on 140 LoCoMo questions.
Critical strengths
The method gives a simple way to adjust evidence-sensitive supervision without answer-level rewards or a learned value model. The convexity argument provides a clear justification for the scalar search. The experiments include matched component ablations and transfer to three benchmarks not used for training. These comparisons provide useful evidence that the approach can improve answering beyond the training benchmark.
Critical weaknesses
The ablations do not establish whether the mirror budget is better than simply weakening the evidence contrast. The clearest calibration gain is on MuSiQue. On the other three benchmarks, the paired Judge confidence intervals cross zero. These intervals also condition on fixed checkpoints. The evidence therefore leaves uncertainty about the benefit of adaptive calibration and its consistency across independent training runs.
Questions for authors
-
How does mirror calibration compare with a fixed coefficient , or an uncalibrated target with tuned on the same validation set? This would help separate the value of prefix-specific calibration from simply using a weaker contrast.
-
Are independent training repeats available for MCC-OPD and the matched uncalibrated baseline? Do the calibration gains persist when uncertainty from training is included, rather than only resampling evaluation examples at fixed checkpoints?
-
Were the benchmark corrections and exclusions finalized before comparing model outputs? Are the revised data and evaluation scripts available? Does the method ranking remain similar on the subset with unchanged questions, references, and evidence?
-
Does an independent check of the answers changed by calibration support the reported Judge gains? A second judge or blinded human assessment would be useful, especially on REALTALK, where F1 and EM decrease.
-
How sensitive are the calibration coefficients and task scores to in the shared token partition? Since the remaining vocabulary is grouped into one tail cell, could the aggregation hide evidence-sensitive changes that would affect the mirror budget?
4 comments
Follow-up: Joint Judge × Subset Check and Full 11-Method Rank Consistency
We jointly varied the evaluation subset and Judge to assess average calibration gains, and separately checked the consistency of all 11 student-method rankings under DeepSeek Judge.
- Joint five-seed check across two Judges and three subsets. Using identical answers, references, rubrics, and subset definitions across DeepSeek and blind MiMo v2.6 Pro, we computed five-seed paired same- MCC−FC macro Judge gains (pp, equal benchmark weights per run):
| Subset (Items) | DeepSeek | MiMo | DeepSeek | MiMo |
|---|---|---|---|---|
| Full set (4,591) | +0.095 | +0.097 | +0.160 | +0.151 |
| Unchanged (4,391) | +0.092 | +0.122 | +0.166 | +0.167 |
| Strict (4,264) | +0.100 | +0.094 | +0.266 | +0.269 |
Unchanged excludes revisions to any recorded field, and Strict further removes items with item-specific Judge guidance (seed-level variation is reported in our first response). All 12 mean gains are positive, and at the average calibration gain widens on the strictest unedited, unguided subset under both independent Judges.
- Consistency of the full 11-method ranking. Holding the submitted main-table checkpoints fixed under DeepSeek Judge (distinct from the five-seed comparison above), we computed Spearman rank correlations against the full-set ordering of all 11 student methods across the four benchmarks:
| Subset | Spearman Range |
|---|---|
| Unchanged | 0.984 – 1.000 |
| Strict | 0.964 – 0.995 |
These near-unity correlations show that MCC retains its #1 standing and the overall ranking of the compared student methods remains highly consistent.
Follow-up before the discussion deadline
Dear Reviewer,
Thank you again for your time and constructive feedback. We would like to follow up on our rebuttal and supplementary response, where we have provided additional experiments, analyses, and clarifications addressing your concerns.
We would greatly appreciate your thoughts on these updates and whether any questions remain. If the additional evidence resolves your main concerns, we would be grateful if you would consider raising your score. We are happy to provide any further clarification that would help your assessment.
Thank you for your continued consideration.
Best regards,
The Authors
I read the rebuttal and appreciate the extensive new experiments, which address some of my questions. However, the gain of mirror calibration over a tuned weaker contrast remains small and within seed variability, so the benefit of the paper's central component is not yet clearly established. I maintain my score.
Response to Reviewer 3bve: Matched Controls, Repeats, Blind Judge & Unchanged-Set Checks
Thank you for your sharp methodological questions. We completed all five checks: validation-selected α / constant-c (+0.208 pp, 5 seeds) and single-run fixed-KL (+1.90 pp) controls, 20 training runs yielding 80 benchmark evaluations, unchanged-item checks, blind second-Judge audits, and 5-seed K-sweeps.
Q1. Mirror calibration vs. weaker contrast (c<1) and fixed KL.
Constant c<1 gives Q(c)∝QF1+cαQC−cα, identical to uncalibrated FC with αeff=cα:
Alongside App. C.1/Table 4 (MuSiQue +2.95/+2.95/+2.76/+2.63 with positive fixed-checkpoint intervals; other 3 Judge intervals cross zero), these controls confirm gains beyond weaker contrast and fixed KL caps.
Q2. Independent training repeats.
MCC/FC × two α values × 5 seeds gives 20 training runs, each evaluated on four benchmarks for 80 evaluations; both same-α settings, plus Q1’s validation-selected pair (+0.208 pp), retain positive 5-seed macro Judge gains:
Judge scores are percentages; macro equally weights four benchmarks, with mean±sample SD across five seeds. Gains are percentage points.
MCC leads in 3/5 runs per α. At α=2, calibration nearly halves seed SD (0.241 vs. 0.459) and improves 3 benchmarks (+0.49 REALTALK, +0.10 MuSiQue, +0.16 LongMemEval, −0.12 LoCoMo; F1/EM/BLEU-1 −0.14/−0.21/−0.18). At α=1.5, MuSiQue improves in 5/5 seeds (+1.18 pp).
Q3. Unchanged examples and data availability.
On the 4,391 strictly unchanged items (also excluding 40 decomposition-edited MuSiQue items), MCC stays #1 among all 11 student methods on all four Judge scores, winning 12/16 metrics, with a +2.316-point macro Judge gain over Full OPD versus +2.318 on the full set. Also excluding item-specific Judge guidance (4,264 items), MCC remains #1 on all four Judge scores, with a +2.528-point macro gain over Full OPD. While some edits followed output inspection, all were source-verified and uniformly applied across methods; data, scripts, and logs are ready for release.
Q4. Blind independent Judge check.
Blind re-evaluation of all 80 results with MiMo v2.6 Pro (fixed rubrics, hidden labels, 2 votes + tie-breaker) gives +0.097/+0.151 macro gains at α=1.5/2, matching DeepSeek’s +0.095/+0.160. On REALTALK changed answers across 5 seeds, newly-correct exceed lost-correct predictions: 123 vs. 99 (+24) at α=1.5 and 130 vs. 96 (+34) at α=2, confirming net semantic correctness gains alongside F1/EM.
Q5. Partition size K and tail cell.
Sweeping K∈{32,64,128} (5 seeds each, α=1.5, separate trajectories) shows stable behavior:
The tail preserves the remaining vocabulary's total probability mass but can hide intra-tail shifts. Activation rates use valid tokens; Prop. 1 guarantees the budget on the chosen partition (not cross-partition invariance or full-vocabulary equivalence). Across tested K, coefficients and activation rates stay close, and K: 64→128 shifts mean accuracy by just +0.13 pp.
Given these results across all five questions, we respectfully invite you to consider updating your score.