- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper talks about literal evidence subsets with exact cardinality and no reweighting. Under additive logit model, the range of the left alone contribution is equal to the largest change in posterior log odds.
The main theoretical result gives the exact minimum and maximum forward KL divergence at a fixed residual range and full posterior. The minimum depends only on the two smallest posterior masses. This result converts a range-solver lower bound into a KL optimality certificate. The paper also proves that the factor is sharp and constructs cases where bounded-exchange refinement cannot reach the better KL solution.
Critical strengths
-
The paper identifies the exact lower endpoint, proves that it is attained, and shows how the two smallest posterior masses control the result.
-
The paper derived when range minimization gives a KL guarantee in addition to showing why local KL refinement can remain trapped, even after exact range initialization and even when several atoms may be exchanged.
Critical weaknesses
-
The real-model experiments do not report the envelope lower bound, the certified KL gap, or the practical tightness of the approximation factor. hence,main theorem is not connected to the main application by a quantitative result.
-
CPC has higher displayed direct KL than LLR norm at all four operating points. Simple lexical selection is best in the atomic setting, while MMR lexical is best in the paired setting. Importantly, all 16 calibrated constructions issue zero useful certificates when evidence is omitted.
Questions for authors
-
Can the proposed KL optimality certificate computed on the controlled experiments using verified MILP lower bounds? How tight is it?
-
Is there any neural setting where CPC gives a useful certificate or a clear benefit over lexical, MMR, or LLR selection?
2 comments
Q2. Is there any neural setting where CPC gives a useful certificate or a clear benefit over lexical, MMR, or LLR selection? (Direct KL performance and certification under omission)
Yes, useful certification occurs in a narrow paired-evidence setting. In the separate source-disjoint evaluation of 500 sources, additive-nominal CPC at calibration level 0.05 issues two joint winner-preservation certificates at each paired item fraction, 0.20 and 0.35. Both targets keep their full-context winner.
Each case retains two of three paired items. The same two sources issue at both fractions, so these are two example-level successes, not four independent ones.
| Setting | Actual omissions among 500 sources | Useful certificates | Observed false issuances |
|---|---|---|---|
| Atomic 0.20 | 397 | 0 | 0 |
| Atomic 0.35 | 397 | 0 | 0 |
| Paired 0.20 | 378 | 2 | 0 |
| Paired 0.35 | 378 | 2 | 0 |
The fractions set the item-budget rule; both certified examples retain two of three paired items at each fraction. These successes are too few to estimate conditional precision, the correctness rate among issued certificates. The initial cohort's zero-issuance result remains separate.
Section 5.1 and Table 4 (pp. 7–8) report all five calibration constructions. Appendix K.2 (pp. 22–23) gives the population and scope.
The direct-KL comparisons support no general CPC advantage over the named selectors. We average both target losses within each of the 500 sources. Lower KL is better. Retained item counts are matched; tokens and runtime are not.
| Setting | CPC | Lexical | MMR | LLR | KL-greedy |
|---|---|---|---|---|---|
| Atomic 0.20 | 1.3403 | 1.0736 | 1.1117 | 1.1663 | 1.2291 |
| Atomic 0.35 | 1.0066 | 0.7323 | 0.7485 | 0.8708 | 1.1052 |
| Paired 0.20 | 0.9971 | 0.8470 | 0.8661 | 0.9839 | 1.0729 |
| Paired 0.35 | 0.9158 | 0.7234 | 0.8032 | 0.8806 | 0.9983 |
The declared Monte Carlo/Holm procedure uses 10,000 paired source-level resamples for all 16 planned comparisons. It favors lexical at atomic 0.35 and paired 0.35, and MMR at atomic 0.35. It supports no CPC improvement over a named comparator; the others remain unresolved.
Table 3 (p. 7) reports the means. Tables 12–13 (pp. 22–23) include all 13 policies and the paired uncertainty analysis.
Thank you again for prompting these measurements of certificate tightness and practical usefulness. We keep the practical claims limited to the measured certificate quality and the two paired-evidence successes.
Thank you for recognizing that the exact KL lower endpoint is attained, that it depends on the two smallest posterior masses, and that range initialization followed by local KL refinement has both guarantees and limits.
We also appreciate your constructive questions about certificate tightness and practical performance on neural models.
We computed the certificate with independently verified range bounds and added exhaustive neural measurements. A separate source-disjoint evaluation tests useful certification and baseline fidelity (Sections 5.1–5.2 and 6.1).
Q1. Can the proposed KL optimality certificate computed on the controlled experiments using verified MILP lower bounds? How tight is it? (Certificate tightness and approximation factor)
Yes. We evaluated it in 72 additive cells with 16 evidence items and five answer options. We independently checked the mixed-integer linear programming (MILP) range bounds against exact enumeration of the stored inputs, then evaluated all 514,512 subsets using outward-rounded KL intervals.
Lower-bound efficiency is the certified KL lower bound divided by the attained KL optimum. The controlled results are:
Median efficiency: 0.105, ranging from about 0.0000015 to 0.8516.
Median certified CPC gap: 0.00552 nats.
Median actual CPC gap: 0.000502 nats.
The certified gap uses the lower bound; actual regret uses the attained optimum. Each is a separate median (Section 6.1, Table 7, pp. 9–10; Appendix J, p. 21).
Every range-minimizing incumbent reaches the exact minimum for the stored inputs. We capped tiny raw-bound overshoots before conversion. The corrected KL lower bound agrees with the exact-range bound to numerical precision.
Most remaining conservatism is geometric: the envelope covers all residuals at a given range, while feasible subsets attain only finitely many. It can include residuals that no feasible subset produces.
We also scored every feasible subset in 264 neural families from 64 RACE, DREAM, and SciFact sources, giving 154,178 candidate scores across both targets.
Among 262 positive-optimum families, median efficiency is 0.173, with interquartile range [0.0536, 0.3596]; 40 families reach at least 0.5. Across all 264 families, median certified and actual CPC gaps are 0.129 and 0.104 nats (Section 5.2, Table 5, p. 8; Appendix K.1, p. 22).
The median reference-specific finite-range factor is 41.4, versus about 2,260 for the mass-only factor. These are approximation bounds, not CPC's achieved ratios. The direct-range initializer's median achieved ratio of 1.00 belongs to the target's range-minimizing subset.
Ratios exclude the two zero-optimum families; absolute gaps retain them.
The neural analysis uses direct target scores for complete subset families. Sources are reused across targets, evidence groupings, and budgets, so these summaries are descriptive. An estimator-only selector would not automatically have this information.
A surrogate lower bound certifies its own family. Transferring that bound to the target optimum requires valid transfer control across the whole target candidate family (Sections 2 and 3.2; Appendix K.1).