acceptodds
Under review as a conference paper at ICLR 2027

When Semantics Stay the Same but Predictions Change: Auditing Checkpoint Portability under Local Nominal Re-serialization

Abstract

Aggregate task scores can conceal substantial changes in a model’s individual de- cisions. We study this reliability gap for conversation-local nominal identifiers, where two preprocessing pipelines may assign different integer names to the same local participant partition without changing the underlying task semantics. Nev- ertheless, a frozen checkpoint using absolute-ID embeddings can produce differ- ent predictions across such semantically equivalent re-serializations. We formalize the required invariance as semantics-preserving checkpoint portability and intro- duce a semantics-gated audit that first verifies preservation of the task-semantic object and then compares paired executions of the same frozen checkpoint at the decision, confidence, and unit-risk levels. The audit distinguishes three in- terface regimes: absolute-ID lookup, empirical mitigation through augmentation or JS consistency, and exact-contract canonical or relation-only representations. On held-out ECF Pair/Query evaluation, absolute-ID lookup produces unit-level portability risks of 0.6407 and 0.6103 while ∆F1 is only −0.0038 and +0.0001, showing that nearly unchanged aggregate performance can coexist with substan- tial prediction-level divergence. Derived-ID MRDA and Entity Relation Classi- fication reproduce the same qualitative pattern across task and identifier settings. Empirical mitigation improves selected portability metrics but does not yield exact portability, whereas canonical and relation-only interfaces achieve exact consis- tency, zero confidence drift, and zero portability risk under their implementation contracts. These results demonstrate that aggregate task performance alone does not certify frozen-checkpoint portability and motivate semantics-aware, interface- level reliability evaluation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
checkpoint portability, semantic invariance, model robustness, representation invariance, nominal identifiers, behavioral evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1ABiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether frozen dialogue models remain behaviorally consistent when local speaker identifiers are renamed without changing their meaning. The authors address their claims by introducing a portability audit and evaluate several speaker-ID interfaces.

Critical strengths

1. The paper studies a question about whether arbitrary local naming can affect model predictions. The portability audit provides a simple and concrete way to expose this kind of representation sensitivity.

Critical weaknesses

1. The broader reliability framing seems overclaimed than what the current evidence supports. The setting only considers local identifiers that carry no task semantics, the experiments mainly test whether the model satisfies this particular interface contract. This is meaningful for the nominal-ID setting, but it can not support a broader claim about speaker-aware reliability

2. The main scientific insight appears fairly limited. Since the local IDs are defined to carry no semantics, prediction changes under renaming mainly reflect sensitivity to an arbitrary representation. The corresponding permutation invariance after removing this arbitrariness also seems fairly expected.

Questions for authors

1. What evidence supports extending the conclusions from nominal local identifiers to broader speaker-aware reliability?

2. For the canonical and relation-only interfaces, what empirical behavior is established beyond the invariance built into their construction?

3. At the individual-prediction level, how frequent are the changes underlying the reported conversation-level unit risk?

4. Which of the paper's conclusions are intended to extend to settings where speaker information carries task-relevant semantics?

Reviewer_u5R1OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what happens when a frozen dialogue model is run through two different preprocessing pipelines that assign different integer IDs to the same speakers while leaving the conversation, the labels, and the who-is-the-same-speaker structure unchanged. The authors define this as semantics-preserving checkpoint portability, add a semantic gate that only admits transformations which genuinely preserve the task object, and compare paired runs at the decision, confidence, and per-conversation level.

I should say up front that this is not my area. I do not work on speaker-aware dialogue modelling, so what follows is an outsider's read, and in particular I cannot judge how common the interface they attack actually is in practice.

Critical strengths
  • I liked the semantic gate. Checking that a transformation actually preserves meaning before calling a prediction change a failure seems like the right order to do things in, and the negative-controls table made it concrete for me.

  • The protocol hygiene is the part I could check most easily, and it looks careful: checkpoint, threshold, and evaluation subset frozen before the held-out run, bootstrapped intervals, recorded checkpoint hashes, and every individual transfer in the appendix rather than only pooled means.

  • If it generalizes, the relation-only interface being both invariant and a little higher F1 than absolute lookup (0.371 vs 0.323 on Pair) looks like a useful practical takeaway — it suggests the fix is not costly.

Critical weaknesses

As noted above, I am not an expert here, so several of these are things I could not resolve from the paper rather than things I am confident are wrong.

  • I could not work out how much of the result is empirical. If a model embeds an arbitrary integer, relabelling it must hit a different embedding row, and the paper itself describes the interface as "serializer-sensitive by construction." The canonical and relation-only interfaces are likewise invariant by the paper's own Eq. 12. So the rows of 1.0000 and zeros read to me like a check that the implementation matches the maths rather than a finding. If I am right about that, the empirical contribution is narrower than the abstract suggests, and it would help to say explicitly which parts could have come out otherwise. If I am wrong, I would like to understand why.

  • I am unsure what the headline number adds. Unit risk counts a conversation as affected if any prediction in it flips. Decision consistency on Pair is 0.949, so roughly 5% of decisions flip; if a conversation holds about 20 candidate pairs then 1 − 0.949²⁰ ≈ 0.64, which is the reported risk. That made me wonder whether unit risk and decision consistency are the same fact at two granularities, with the larger-looking number partly reflecting how many predictions a unit contains. The paper does not report predictions per unit, so I could not check. The MRDA value of 1.00 over 12 dialogues looked like the same pattern at its limit.

  • The practical importance is the thing I most wanted the paper to settle, and I could not settle it. How many speaker-aware models in use today embed the integer ID rather than putting speaker names into the text? My outsider impression is that text-based speaker marking is now common, which would not fail in this way, but I genuinely do not know. One widely used public checkpoint shown to be affected would make the case for me.

  • The experimental base looked small to me, and one setting I could not follow at all. The main held-out result uses two seeds, MRDA uses 12 dialogues, and the Entity Relation Classification setting appears in Table 1 with no interval and is never described — I could not tell which dataset it is or how many groups it contains. The F1 values in the ablation are also low (0.32 to 0.43), and I was unable to judge whether these checkpoints are strong enough for their instability to tell us much about well-trained models.

Questions for authors

see Weaknesses

Reviewer_XinFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a frozen checkpoint keeps its per-instance predictions when conversation-local speaker IDs are re-serialized by a different deterministic scheme (first-appearance S1, lexicographic S2, SHA-256 S3) that preserves the speaker-equality partition, text, order, labels and candidate keys. It defines "semantics-preserving checkpoint portability" (Eq. 4), a "semantic gate" that admits only such bijective renamings (Eq. 3), and a set of paired-execution metrics: exact decision consistency, Jaccard, confidence drift, correct-to-incorrect rate, unit-level risk (fraction of conversations/dialogues with at least one changed decision) and ΔF1. Five interfaces are compared: absolute-ID embedding lookup (I), augmentation (I_aug), JS-consistency (I_JS), a canonicalization wrapper around I (C), and a relation-only encoder (R). On held-out ECF Pair/Query (250 conversations, 2 seeds, 6 transfer directions) the I interface shows exact consistency 0.949/0.974 and unit risk 0.64/0.61 with ΔF1 ≈ 0 (Table 1); MRDA (12 dialogues) and an unspecified Entity Relation Classification setting show the same pattern; C and R show exact consistency 1.0 and zero risk in every row. A dev-set ablation (Table 2) shows I_aug/I_JS reduce but do not eliminate flips.

Critical strengths

The experimental bookkeeping is consistent and auditable. Recomputing from Tables S1–S3 and S6–S7 reproduces every pooled number in Table 1, Table 2 and §4.2–4.4 (e.g., Pair risk 0.640667, dev C→I sums 3,873/2,175, ΔF1 −0.001135/+0.001374); held-out risks are exact multiples of 1/500, matching 250 units × 2 seeds. This matters because it means the reported effect is real and not a reporting error, so the assessment can focus on what the effect means.

Critical weaknesses
  1. The "exact-contract" result is tautological, and the paper presents it as an empirical finding. C "reuses the matching frozen I model state through a canonicalization wrapper" (p.5), i.e., it re-encodes the deployment view with the training serializer before calling the same I model; Table 2 confirms C's F1 equals I's (0.3229, 0.3942). Under Eq. 12 the two inputs are then bit-identical, so exact consistency 1.0 and zero risk/drift follow from determinism, not from any experiment. Yet Table S4 (48 rows), the C/R rows in Tables 1/S1/S2, Figure 3's five C/R points, and the abstract's "Canonical and relation-only interfaces achieve exact consistency, zero drift, and zero portability risk" all report f(x)=f(x) as a result. The non-trivial question—what R costs or gains in held-out task performance—is never answered, since Table 1 reports only ΔF1 and no absolute F1 for I/C/R. This leaves the "interface-level reliability comparison" contribution without content and is the main reason I do not see a sufficient contribution.

  2. The headline unit-level risk is largely a function of unit size and of units whose IDs do not change, so the main numbers overstate the finding, and the setup needed to interpret them is missing. Eq. 19 counts a conversation as "at risk" if any aligned candidate flips; instance-level disagreement is 5.1% (Pair) and 2.6% (Query), and with ~20 candidates per conversation 1−0.949^20 ≈ 0.65 already matches 0.6407. For MRDA, dialogues contain on the order of a thousand utterances, so "all 12 dialogues change in every bootstrap resample" (p.7) is arithmetically forced by the 10.6–11.5% utterance-level disagreement. Conversely, the paper reports that S2 changes only 76 of 109 dev ID sequences (p.8), so ~30% of units have an identity transformation and cannot flip, which is never conditioned on. No model architecture, encoder, training configuration, λ_JS, or augmentation views are given for any setting; MRDA has no citation and uses "12 held-out dialogues" (standard test split: 11 meetings); "Entity Relation Classification" has no source, size or model; and the "supplementary material" and JSON/Markdown "companion files" promised in the Reproducibility Statement (p.9) and Appendix B were not submitted. Without these, the 0.64/0.61 headline cannot be interpreted or reproduced, which determines a reject recommendation.

Questions for authors
  1. For each setting, what are the model architecture, encoder, training configuration, λ_JS and the augmentation views used for I_aug, and how do the "20 checkpoints" (p.6) decompose across families, seeds, source serializers and interfaces?
  2. How does interface C differ from "always apply the training serializer at deployment", and what are the absolute held-out F1 values of I, C and R (Pair and Query)?
  3. What is the mean number of aligned instances per unit in each setting, and what is unit-level risk restricted to units whose ID sequence actually differs between source and deployment views? For MRDA lexical (risk 11/12), does the unchanged dialogue have identical S1 and S2 sequences?
  4. What are the source, split and citation for the MRDA "12 held-out dialogues" and for the Entity Relation Classification data (number of groups, instances, model)?
  5. Can the promised companion files (heldout_serializer_portability_rows.json, per-seed rows with bootstrap intervals, threshold-sweep table) be provided, and what statistic does the 2,000-resample bootstrap in Table 1 resample (a single transfer row or the 12-row mean)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.