- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether model merging preserves error detectability (the correctness AUROC of confidence scores), not only accuracy. To separate confidence changes from label changes, it compares experts and merged models on agreement subsets, where predictions and correctness labels coincide. On five CLIP ViT-B/32 tasks, experts reach 86.4% macro agreement-subset AUROC versus 69.2% for TIES. A three-class linear construction shows that parameter averaging can fully reverse a perfect error ranking without changing any prediction, through an example-dependent Jensen gap in the competing-class log-sum-exp. The paper then proposes PED, a post-merge correction in a 24-dimensional task-vector subspace. PED minimizes a pairwise ranking surrogate under margin floors on correctly predicted fitting examples. It keeps only candidates whose macro fixed-label fitting AUROC is at least the base’s. With 128 fitting and 128 selection labels per task, PED raises TIES' AUROC from 84.7% to 88.2% and accuracy by 1.7 points, outperforming LMN and ProbSurgery. It also recovers 72.1% of the expert–TIES gap on fixed base decisions and gains 3.9–4.2 AUROC points on ViT-L/14.
Critical strengths
- The agreement-subset evaluation separates changes in confidence ranking from changes in which examples are classified correctly. The three-class setup takes this further by showing that rankings can change even if all predictions stay the same. This is an important idea, since typical model-merging evaluations focus only on accuracy and might miss this type of issue.
- PED outperforms TIES in both AUROC and accuracy, and the correction baselines used in the main experiments. These improvements remain consistent on ViT-L/14 and with task arithmetic. The fixed-label experiment shows that the gains are not just due to different predictions. Since the correction is built directly into the merged weights, there is no need for an inference-time module.
Critical weaknesses
- The experiments cover eight image-classification tasks, two CLIP backbones, and two merging methods. It remains unclear how well the method generalizes to other architectures, domains, or tasks. More importantly, fitting and selection examples are drawn from the official held-out evaluation data, although they are kept disjoint from the audit examples. This is a valid separated evaluation protocol, but it leaves open how practical PED would be when labeled examples from the target deployment distribution are scarce or unavailable.
- The evidence does not fully isolate the source of PED’s gains. The controls suggest that the complete constrained optimization procedure is effective, but the paper acknowledges that the separate contributions of the optimizer, margin constraints, and eligibility test are not identified. In addition, although the linear analysis identifies a precise mechanism for ranking reversal, the nonlinear decomposition includes a residual whose empirical role remains unresolved. These limitations weaken the mechanistic explanation for the observed improvements, though they do not invalidate the reported results.
Questions for authors
- Since fitting and selection labels are drawn from the official test split, can the authors clarify how the method would be deployed when target-task labels are available only from a separate validation set? Do the gains persist when adaptation uses a distinct training or validation source and the final audit comes from a separate evaluation split?
- Can the authors isolate the contributions of the protected-margin constraints and the fitting-AUROC eligibility test from the choice of optimizer? In particular, how much of the improvement persists when ranking optimization, preservation constraints, and candidate selection are changed individually while keeping the coefficient space and data budget fixed?
- Can the authors quantify how much of the observed ranking degradation in the nonlinear models is associated with the Jensen-gap term versus the residual between weight merging and the reference logit ensemble? This would help establish how directly the linear explanation accounts for the empirical phenomenon.
- (Not affecting my rating) Is there a reason why AdaMerging is not included in the reporting?
My rating is most likely to be affected by a strong answer to question 2.