acceptodds
Under review as a conference paper at ICLR 2027

Privacy Noise Masks Then Reveals Order and Partition Effects in Sample-and-Aggregate ICL

Abstract

In sample-and-aggregate private in-context learning (ICL) for classification, privacy noise changes not only the amount but also the source of released-label instability. Using a full-domain Gumbel/exponential count channel, we instantiate a law-of-total-variance identity, valid for any count-based categorical release kernel, that separates released Gini exactly into release randomness, within-partition order variation, and partition variation. Under tight Gumbel privacy, a near-uniform release masks order and partition structure. In three sampled designs, grid-interpolated crossovers lie between and , so stability near the commonly evaluated need not imply weak upstream structure. Gaussian report-noisy-max on the same counts shows the same qualitative masking, without implying cross-mechanism epsilon equivalence. Nested and matched designs attribute most aggregate upstream variation to ordering and reveal task-dependent partition concentration. Signed multiclass margins prospectively detect nonzero downstream order activity on TREC question classification, including a weaker 14B replication, but do not universally rank vote strength or severity. A prospectively frozen sizing forecast predicts an unseen Banking77 configuration within 0.007 on accuracy and 0.002 on fidelity. Holding 72 private records fixed, 24 three-example voters preserve similar pre-release plurality accuracy yet achieve substantially higher released utility than eight nine-example voters, exposing a condition-specific transmission window. Low measured order components under tight privacy can therefore be channel artifacts. Internal, privacy-accounted audits of pre-noise votes—or analyses on public development data—should accompany released-label stability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, in-context learning, large language models, sample-and-aggregate, order sensitivity, prediction instability, exponential mechanism

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qzNkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

There are several sources of instability that arise from sample-and-aggregate classification ICL. The ordering, partition, and privacy noise all affect the classification output. The paper investigates how increasing privacy noise masks the instability arising from the other two (ordering and partition).

The paper uses a sampled demonstration set that is partitioned and fed as a context to a frozen LLM. This produces vote counts for each class in the classification task. Afterwards, the final classification is released by a Gumbel report-noisy-max (RNM) mechanism, which is an exponential-mechanism variant with a closed-form distribution. The tight Gumbel privacy masks upstream order and partition effects. Under strong privacy, the output distribution of this mechanism approaches the uniform distribution.

Critical strengths
  • The explanatory framework for the privacy masking of the upstream order and partition effects is interesting.
  • The explanatory framework is well supported by empirical evidence, and the diversity of models and datasets are reasonable.
  • The authors performed careful evaluation audits including tie-floor and exact order reconstruction checks
  • The paper also presents interesting experiments showing the utility advantage of using more smaller voters at intermediate privacy levels.
Critical weaknesses
  • The explanatory framework is novel; however, the mechanisms used are not novel, and the paper only considers one type of ICL which is LLMs. For example, tabular foundation models might be sensitive to partitioning but a permutation-invariant architecture might not be sensitive to the ordering of samples within each partition. For these reasons, I believe that the paper's contributions are limited.
  • Proposition 2 decomposes the total prediction instability by first conditionon on Π\Pi, then ordering conditioned on the partition O∣ΠO\mid \Pi. However, section 3 describes the pipeline as proceeding from ordering OO to partitioning Π∣O\Pi \mid O, and then to private release. Both yield valid decompositions through the law of total variance but they can assign different contributions to ordering and partitioning.
  • The fixed-record experiment demonstrating the transmission window uses only one model-dataset combination, providing limited evidence that this phenomenon generalizes.
Questions for authors
  • Would the same explanatory framework apply to other ICL methods, such as permutation-invariant tabular foundation models?
  • Can the authors clarify why this particular decomposition is used and why it is not consistently described in the paper? Also, whether using one over the other would affect the paper's conclusions?
  • Would the transmission window conclusion generalize beyond the single model-dataset combination?
4 comments

Thank you for identifying the inconsistency between the pipeline description and the conditioning order. We have corrected the exposition and made the attribution convention explicit.

Why partition first, and what changes under reversal? The private pipeline partitions the pool, orders examples within each subset, collects one vote per subset, and releases a private label. The revised setup and Figure 1 caption state this order. Section 3.1 is now “Full-context diagnostic: signed margins”; that diagnostic is separate from the voting pipeline.

For the same joint law of partition Π, order O and release probabilities q, reversing the conditioning leaves two quantities unchanged:

  • the privacy term, E[1 − ‖q‖²];
  • total upstream variation, E‖q − E q‖².

Consequently their crossover is unchanged. The attribution of upstream variation to order versus partition can change. With independent population randomizers, write the upstream variance as A_Π + A_O + I, where A_Π and A_O are main effects and I is their interaction. Partition-first attribution gives R_part = A_Π and R_ord = A_O + I; reversing the conditioning gives R_ord′ = A_O and R_part′ = A_Π + I (Appendix D.1, Lemma 4).

The sampler coupling matters. Our pool is uniformly shuffled and sliced into ordered blocks, represented by Π. An independent O specifies a position permutation within each block. Fixing O maps the uniform whole-pool shuffle bijectively onto another uniform shuffle. Thus E[q | O] = E q and the population order main effect A_O is zero under this coupling. Intuitively, a position permutation acts on different examples in different partitions. Sorting each block's initial arrangement can break this symmetry; the appendix includes a toy counterexample.

This is not a statement that order-sensitive models have no order effect. Partition-first R_ord measures variation when membership is held fixed and the examples are reordered. It includes the interaction and answers the fixed-subset question we study. An order-first attribution answers a different question and, under this coupling, assigns the interaction to partition instead. Our order-dominance claims therefore concern the partition-first estimand, not a convention-independent ranking of the two factors.

The zero-main-effect statement is about the population sampler, not a finite empirical estimate. The logged schedules are generated independently for different partitions and subsets; shared schedule indices do not form a crossed partition-by-order table. We do not treat them as such.

Where this is now stated. Section 3 opens with the sampling order and attribution convention (p. 3). “Why partition first?” (§3.4, p. 6) explains the choice. Appendix D.1 gives the reverse-conditioning formulas and exact toy enumeration. The privacy term, total-upstream conclusions and crossover survive reversal; the separate order/partition shares are explicitly convention-dependent.

Applicability beyond LLM-based ICL. The attribution identity is not specific to a language model. It applies to categorical releases from internal vote counts under the stated execution model. For a predictor whose votes are exactly invariant to demonstration order, R_ord = 0 and G = R_priv + R_part. Partitioning can still matter. Retaining the same full-domain Gumbel channel also retains the tight-privacy limit and count-gap bounds; a finite crossover depends on partition variation and the tie floor. This theoretical applicability is distinct from empirical validation on tabular foundation models, which we have not performed (§3.4, p. 6).

We agree that the release mechanism itself is established. The contribution is the attribution analysis and the measured finite-budget consequences for LLM-based sample-and-aggregate classification, not a new private-ICL mechanism.

Evidence beyond the original transmission-window condition. To test the fixed-72-record result on a second model, we evaluated Mistral-7B-v0.3 on Banking77 during discussion. It uses the same 100 queries, labels and ordered 72-record pools as the Qwen comparison, five partitions, and the same endpoints and paired query bootstrap. No pass/fail threshold was set for this run.

At ε = 1, replacing eight nine-example voters with 24 three-example voters gives:

EndpointEight voters24 votersDifference [95% interval]
Released accuracy.222.542+.320 [.256, .384]
Plurality fidelity.301.960+.660 [.639, .681]
Pre-release plurality accuracy.602.545−.056 [−.098, −.021]

Pre-release accuracy is a secondary endpoint. Its decrease shows that the released-accuracy gain does not require higher average plurality accuracy before noise (§5.3, p. 9; Table 18).

The curve shape does not replicate. In the post-hoc Mistral replay, the accuracy difference is −.014 [−.049, .017] at ε = 2 and −.058 [−.100, −.023] at ε = 4 (pointwise intervals; Table 19). Qwen's advantage instead becomes small at the deterministic reference. We therefore use “transmission window” for budgets where more voters improve mean released accuracy, without asserting a common shape or an inferentially established window boundary. The 72-record result now covers two models, still on one task.

Complementary cross-task evidence. The separately frozen n = 24 panel covers Mistral-7B and Qwen2.5-1.5B on DBpedia and TREC-fine. Each subset casts its most frequent prediction across all within-subset orders, so this is a different voting protocol. Comparing (S, s) = (24, 1) with (8, 3), released accuracy increases at ε = 1 in all four conditions (+.036 to +.139), then decreases by ε = 8 in post-hoc replay (Table 3). TREC-fine endpoints were primary; DBpedia was descriptive. This supports budget-dependent allocation trade-offs across tasks, but is not a replication of the Banking77 curve. Section 5.3 and §7 now state these distinctions directly.

Thank you for the detailed responses and revisions. I appreciate the effort invested in addressing my concerns.

I believe that the authors have addressed the conditioning-order issue and strengthened the empirical evidence.

However, my main concern remains the significance of the central contribution. The observation that stronger privacy masks upstream variation follows directly from the behavior of the well-known exponential mechanism.

Therefore, while I acknowledge the improvements in the revisions, I will maintain my original rating of 2.

Thank you for reading the revision and for confirming that the conditioning-order issue is resolved. We agree that masking as ε → 0 follows directly from the exponential mechanism. Our intended contribution is the empirical characterization of configuration choices at finite privacy budgets. With the same 72 private records and ε = 1, switching from 8 voters with 9 demonstrations each to 24 voters with 3 demonstrations each increased released accuracy by 37.9 percentage points for Qwen2.5-7B and 32.0 points for Mistral-7B. In the Mistral experiment, this improvement occurred despite lower pre-noise plurality accuracy. The limiting argument alone does not determine the magnitude or practical consequences of this trade-off. We respect your assessment of its significance and appreciate your constructive review.

Reviewer_UEkwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the effects of demonstration order, data partitioning, and privacy noise on prediction instability in sample-and-aggregate in-context learning. It introduces a diagnostic framework based on a hierarchical decomposition of categorical disagreement, separating variation associated with private release, within-partition ordering, and partitioning. For the Gumbel/exponential release mechanism, the analysis characterizes how strong privacy noise suppresses the order and partition components while overall output randomness remains high. The paper also examines margin-based diagnostics and the effects of allocating a fixed private dataset across different numbers of voters. The analysis is supported by nested experiments, exhaustive reconstruction of independent subset-order combinations, and fixed-record allocation comparisons that measure pre-release accuracy, release fidelity, and released accuracy across privacy budgets.

Critical strengths
  1. The paper offers a useful framework for interpreting instability in private ICL.
    Previous studies have established both prompt-order sensitivity and privacy-preserving voting. This work connects these topics by examining how much of the variation in released predictions is associated with demonstration order, partitioning, and release noise. A particularly useful distinction is that a reduced contribution from order variation does not necessarily imply more consistent predictions: the release mechanism may simply be introducing more randomness.

  2. The empirical conclusions are supported by careful checks and transparent reporting.
    The authors reconstruct all 686^8 independent combinations of subset permutations to investigate whether the apparent dominance of order effects is an artifact of synchronized ordering. The estimated order-share intervals remain above one half across the five matched conditions. The treatment of tied votes, reporting of unsuccessful interventions, and acknowledgment of the limitations of the crossover bootstrap further strengthen confidence in the conclusions within their stated scope.

  3. The fixed-data comparison provides convincing evidence that allocation affects performance through the release mechanism.
    With the private pool held at 72 records, the difference in pre-release plurality accuracy is approximately 0.8 percentage points, whereas the improvement in released accuracy reaches approximately 37.9 percentage points at ϵ=1\epsilon=1. This helps distinguish the observed effect from a simple benefit of using more private data. The reversal of accuracy gains at larger privacy budgets in a separate experiment also clarifies that the allocation recommendation is conditional.

Critical weaknesses
  1. The methodological contribution is incremental, making the empirical findings particularly important.
    Prompt-order sensitivity, sample-and-aggregate prediction, private voting, and the exponential mechanism all have established precedents. The variance decomposition and its strong-noise limiting behavior also follow from standard mathematical tools. The paper would therefore benefit from a clearer separation between established ingredients, consequences derived from those ingredients, and empirical findings that provide additional insight.

  2. The evidence for the main structural conclusions covers a relatively restricted set of configurations.
    The principal crossover results are based on three small nested designs, while the exhaustive independent-order analysis uses only three demonstrations per subset. The connection between full-context margins and voting behavior is also established mainly for synchronized deterministic activity. Consequently, the current evidence does not establish how reliably order dominance or specific crossover values transfer to larger subsets, other models, or different execution protocols.

  3. The practical recommendations remain dependent on the task and privacy regime.
    Increasing the number of voters while reducing the information available to each voter involves a familiar ensemble trade-off. The strongest fixed-record result concerns one Banking77 configuration, the broader experiments show reversed accuracy gains at ϵ=8\epsilon=8, and the evaluated selectors do not outperform simply choosing the largest ensemble. Although a deployment algorithm is not required for this diagnostic study, these findings limit how broadly its configuration advice can be applied.

Questions for authors
  1. What is the central empirical finding that goes beyond the existing literature?
    Relative to prior work on order sensitivity, PATE, and DP-ICL, which result provides the strongest additional insight? Please distinguish this result from the standard observation that sufficiently strong noise makes the release distribution nearly uniform, and identify the experiment that directly supports it.

  2. Does order variation remain dominant when each subset contains more than three demonstrations?
    Are there existing sensitivity results using independently sampled permutations within larger subsets? Please also clarify which crossover intervals condition on the sampled partitions and order schedules, and whether any analysis accounts for resampling these upstream factors. Exhaustive enumeration is not necessary, but additional evidence or a precise statement of the limitation would help.

  3. What concrete allocation guidance follows from the fixed-record experiments?
    Beyond the general trade-off between voter count and individual voter quality, what decisions do the results support? Please distinguish recommendations for release fidelity from those for predictive accuracy and specify the relevant privacy-budget ranges. If applying the guidance requires access to private counts or margins, what additional privacy accounting would be necessary?

3 comments

Thank you for the precise questions. We have clarified the central empirical finding and separated it from the standard mathematical ingredients.

1. Strongest additional insight and its direct evidence. At fixed records and ε, similar or even lower pre-release plurality accuracy can coexist with substantially higher released accuracy; as ε changes, the preferred allocation can reverse. The direct test is the fixed-72-record comparison in §5.3, not the ε → 0 argument.

In the frozen Qwen2.5-7B/Banking77 test, eight nine-example voters and 24 three-example voters use the same records for each of 100 queries. At ε = 1, the 24-voter arm changes pre-release plurality accuracy by +.008 [−.032, .050], but released accuracy by +.379 [.318, .441] and fidelity by +.636 [.619, .653]. The first interval does not establish equivalence. The second-model Mistral check added during discussion gives higher released accuracy (+.320 [.256, .384]) despite lower pre-release accuracy (−.056 [−.098, −.021]); its post-hoc ranking reverses as privacy relaxes (Tables 18–19).

The familiar teacher-count trade-off is acknowledged through PATE (§5.3). In this ICL setting, splitting the pool costs each voter demonstrations rather than training examples. The additional evidence identifies how that trade-off interacts with the release channel, including cases where judging an allocation by pre-release accuracy gives the opposite ordering from released accuracy. It does not establish a universally optimal voter count.

Established tools, derived consequences, measured findings. The revised introduction makes this separation explicit (p. 2):

RoleContent
EstablishedOrder-sensitive ICL; private voting and the exponential mechanism; PATE's ensemble trade-off; total variance and Cantelli's inequality
Derived in this settingPartition-first attribution of categorical disagreement; crossover existence with ties; count-gap and competitor-mass characterization
MeasuredFinite-budget attribution and crossovers; fixed-record utility contrasts and ranking reversals; a frozen sizing forecast; the scoped clean-margin diagnostic

The decomposition makes the finite-budget operating point measurable. In the three nested conditions, release randomness at ε = 1 is .154–.869, versus .002–.034 for total upstream variation (Table 1). The logged-grid crossovers occur at ε ≈ 3.2–4.2. These are configuration-specific findings, not consequences of the limit alone. In particular, small released order components should not be read as an intrinsically order-invariant predictor.

The framework also supported a frozen forecast for an unseen 14-voter Banking77 configuration: accuracy/fidelity .543/.807 predicted, .550/.808 observed (§5.2). This is a validated instance under the stated transfer approximation, not a general scaling law. The revised framing places these empirical consequences, rather than novelty of the underlying inequalities, at the center of the contribution.

2. Larger subsets and the crossover inference target.

Frozen s = 4 test. We extended the two TREC-coarse matched conditions (Qwen2.5-7B and Pythia-6.9B), retaining the 100 queries and expanding each 24-record pool to 32 records. With eight voters and five partitions, all 4! orders are evaluated within each subset and independent joint orders are reconstructed exactly. At ε = 1, corrected partition-first order shares are:

ModelShare [two-sided 95% interval]One-sided 95% lower bound
Qwen2.5-7B.555 [.510, .598].518
Pythia-6.9B.537 [.491, .583].499

The frozen joint criterion requires both lower bounds to exceed .5. It is not supported. Both point estimates exceed one half, but we do not claim dominance at s = 4. The secondary uniform-random-tie zero-noise analysis clears the criterion; it does not change the primary verdict. Because pool size grows with s, this is not a fixed-record comparison or an inferential s = 3 versus s = 4 test (§5.4, p. 10; Appendix D.3).

Post-hoc s = 9 audits. The two Banking77 fixed-record logs provide five partitions and six sampled schedules each, not all 9! orders. With Eq. (14)'s finite-design correction, order shares at ε = 1 are .773 [.697, .847] for Qwen and .643 [.589, .700] for Mistral (uncorrected .694/.591). Query-bootstrap intervals condition on the logged designs and correction assumptions. These indicate substantial within-partition order variation in two non-exhaustive designs; they supplement rather than replace the frozen verdict. Section 7 keeps the established dominance claim scoped to the audited s = 3 conditions.

Crossover ranges. The reported crossover ranges resample queries while holding each query's sampled P = T = 5 grid fixed. They do not account for resampling upstream factors. The synthetic finite-design audit evaluates population coverage across independently generated designs and found that query-only ranges covered the population crossover in none of 40 populations. Corrected point roots shift by −.16 to −.20 (Table 14), but have no validated population intervals. We state this limitation explicitly (§5.1; Appendix C.8), rather than interpret conditional ranges as population uncertainty.

The margin bridge is also target-specific: its prospective detection endpoint concerns deterministic plurality flips under logged schedules. Independent-order and finite-ε targets remain post-hoc sensitivity analyses, not equivalent validation targets (Table 16).

3. Concrete allocation guidance.

EvidenceReleased accuracyPlurality fidelity
ε = 1, n = 72, Banking77, two models24 voters outperform eight (+.379/+.320)Higher with 24 voters (+.636/+.660)
ε = 1, n = 24 consensus panel, four conditions(24, 1) outperforms (8, 3), +.036 to +.139Higher by +.087 to +.272
ε = 8, same n = 24 counts, post-hoc replayRanking reverses in all four conditions, −.055 to −.134Still favors (24, 1) on TREC-fine

The Qwen n = 72 endpoints were frozen; Mistral was added during discussion without a pass/fail threshold. In the frozen n = 24 panel, TREC-fine was primary and DBpedia descriptive, and the voting protocol differs from n = 72.

These results support three decisions, rather than a universal allocation rule. First, use S ≥ S_min from Eq. (10) to screen whether a target fidelity is even attainable by any count vector; passing is not a guarantee for the realized votes. Second, compare released accuracy as well as fidelity, since they can favor different allocations. Third, validate at the intended ε, since the ranking can change with budget. The evaluated plug-in selectors did not outperform maximum S (Table 10), and TREC-fine absolute accuracy remains low, below the post-hoc constant references (§7; Appendix D.4).

Validation can use public development queries with a public demonstration pool. Public queries alone do not make access to private exemplars free. Selecting an allocation from private counts, gaps or margins requires additional accounting, such as a private selection step composed with the release. We provide no such deployable private selector (§6; Appendix C.5).

Thank you for the clarifications and additional experiments. The response has addressed my concerns. I will keep my original rating. Good luck!

Reviewer_W1UhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

this paper operates in the "sample-and-aggregate" ICL setting of wu et al. 2024. this works as follows: a private pool of nn exemplars are split into SS disjoint subsets of size ss. each subset is used as a demonstration set for an ICL call for some LLM (weights frozen, just inference). SS number of votes form a vector of votes and a label is released via standard exponential/gumbel mechanism based on this voting vector: qk∝exp⁡(ϵCk/2)q_k \propto \exp(\epsilon C_k / 2).

there are three sources of randomness in the label: the demonstration order within subsets, the partition of the pool, and the privacy noise.

the authors make the following point: these sources of randomness are not additive. in other words, you can write an index (the Gini instability index i guess) as G=1−∑kpk2G = 1 - \sum_k p_k^2, which is exactly the law of tot. var. , or G=Rpriv+Rord+RpartG = \mathcal{R}_{priv} + \mathcal{R}_{ord} + \mathcal{R}_{part}. at small ε\varepsilon, this is near uniform, so the point the authors make is that the order and partition terms go to zero regardless of how order-sensitive the model actually is.

in the authors experiments, the "upstream terms" only dominate the privacy noise at medium epsilons (like ϵ∗≈3−4\epsilon^* \approx 3-4. the authors have some other experiments that seem to poke at this observation from other angles

Critical strengths

i actually think the conceptual point is reasonable and have not seen it pointed out in the DP-ICL literature previously. that is, that the released label sensitivity under privacy is just because the noise is dominant, so its not really evidence of "robustness"

the experiments are reasonable in their setup and the quality of the ablations / modifications seems good

there are some interesting design recommendations buried in here for folks doing DP-ICL.

Critical weaknesses

i mean, the headline finding is a basic observation in my opinion. if you write down the mechanism definition, you see qk∝exp⁡(ϵCk/2)q_k \propto \exp(\epsilon C_k / 2) is uniform as ϵ\epsilon goes to 0 for any CC, so the idea that tight privacy in some sense masks variance of the other terms is obvious. the additional theory is also basic, just writing down variance properties and cantellis inequality, and a rearrangement of the exp mechanism error bound.

so there is no algorithmic or theoretical contribution, and no "depth" to the logic behind the headline observation. perhaps this would be saved by the strength of the empirical results, but...

the presentation is just actively obstructive to reading the eval / udnerstanding the paper. the abstract is unreadable. what does "synchronized deterministic plurality-order activity" mean?! the contribution framing is odd and non-standard. its really hard to parse much of the text that discusses the experimental results. i found myself having to reconstruct the problem statement myself from scratch, like a detective.

Questions for authors
  1. can you justify the theory beyond the basic stuff i noticed? did I miss anything?
  2. theres maybe something in this " fixed-record" transmission window, but the idea was not clear to me. could you clarify and expand on that result?
  3. i think much of this paper needs to be rewritten. why should we trust you to do this before the camera ready deadline, instead of asking you to make the effort and resubmit?
2 comments

Thank you for identifying both the contribution-framing and readability problems. We agree that the uniform-release limit and the underlying inequalities are standard. Our original presentation did not make the finite-budget question sufficiently clear.

The fixed-record result. Which allocation gives higher released accuracy when the private records and ε are held fixed? We compare eight voters with nine demonstrations each against 24 voters with three each, using the same 72 records per query and 100 Banking77 queries. At ε = 1, the differences below are for 24 voters minus eight; brackets are paired query-bootstrap 95% intervals (Table 18).

ModelPre-release plurality accuracyReleased accuracy
Qwen2.5-7B+.008 [−.032, .050]+.379 [.318, .441]
Mistral-7B−.056 [−.098, −.021]+.320 [.256, .384]

The Qwen comparison is from the frozen protocol; its small mean pre-release difference is not an equivalence result. The Mistral check was added during discussion and uses the same queries and pools. It shows higher released accuracy despite lower pre-release accuracy. A post-hoc replay then reverses Mistral's accuracy ranking as privacy relaxes (Table 19). We call the budgets where more voters improve released accuracy a condition-specific transmission window, not a universal curve. The 72-record evidence covers two models on one task.

Role of the theory. We do not claim a new privacy mechanism or a new general variance theorem. The contribution is a computable attribution framework and its finite-budget empirical consequences. It separates release randomness from order and partition variation, while the gap/competitor-mass analysis explains how the channel transmits a given vote profile. The uniform limit alone neither quantifies the effects above nor selects the more accurate allocation at a finite budget. The practical lesson is to evaluate released accuracy at the intended ε, rather than select an allocation from pre-release accuracy or fidelity alone.

The rewrite is completed, not deferred. The uploaded PDF has a plain-language abstract (p. 1), a problem setup before notation and a clearer §3 title (p. 3), and a direct account of the fixed-record experiment (§5.3, pp. 8–9). Each Results subsection starts with its finding; the distinction between established tools, derived consequences and empirical findings is explicit on p. 2. These changes are available for evaluation during discussion, with the new analyses and corrections listed in the change log.

Thanks. I will maintain my original score, for the stated reasons.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Changes in the revised manuscript

The revision retains the raw experimental records and the pre-specified rules for the frozen analyses. Discussion-period additions are distinguished from submitted evidence and from post-hoc checks.

Additional analyses and evidence

  • Conditioning order (§3.4; Appendix D.1). Reverse-conditioning formulas, the uniform-shuffle coupling argument and an exact toy enumeration explain why total upstream variation is invariant but the order/partition split is not.
  • Second-model fixed-record comparison (§5.3; Appendix D.2, Tables 18–19). Mistral-7B/Banking77 uses the original queries and 72-record pools. We report paired intervals, including the secondary decrease in pre-release accuracy, and a separately labeled post-hoc budget replay. No pass/fail threshold was set for this run.
  • Larger subsets (§5.4; Appendix D.3). The frozen s = 4 joint dominance claim is not supported. The s = 9 finite-design-corrected audits are post hoc and non-exhaustive, not replacement evidence for that verdict.
  • Utility references (Appendix D.4). TREC-fine zero-shot and constant-label references are reported with their scorer-specific and post-hoc limitations. Section 7 notes that private accuracy is below the constant references, which are not independently selected deployment baselines.

Correction to Figure 5 (p. 9)

The logged deterministic plurality point uses fixed tie-breaking. It is now shown as a detached reference rather than joined to the finite-ε curve, whose limit randomizes uniformly among tied maxima. The ε = 1 results and Table 3 are unaffected. Figure 3 already separated its deterministic reference in the submission.

Presentation and scope

LocationRevision
Abstract and §1, pp. 1–2Plain-language abstract; three contributions; separation of established tools, derived consequences and measured findings
Figure 1 caption and §3, pp. 2–3Partition-first pipeline stated; full-context margin identified as a separate diagnostic; §3 retitled “Attributing Prediction Instability”
§3.4, p. 6Explanation of the attribution convention and applicability to order-invariant predictors
§4, pp. 6–7Experiments organized around three research questions
§5, pp. 7–10Finding-led subsections; fixed-record comparison and its condition-specific transmission window explained directly
§5.3, pp. 8–9Second-model result and the separate n = 24 consensus-vote panel, with their protocol differences
§7, p. 10Empirical scope, the unsupported s = 4 joint claim, conditional inference and private-calibration requirements consolidated
AI use, p. 11Claude Code added alongside OpenAI Codex; remaining disclosure unchanged
Terminology and typesettingConsistent uniform-random-tie terminology; detached deterministic reference wording; mathematical minus signs in negative interval endpoints

The revised paper remains 10 main-text pages. Appendix D (pp. 33–34) collects the discussion-period evidence and its provenance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.