acceptodds
Under review as a conference paper at ICLR 2027

Better Agents, Worse Teams: When Low-Order Tests Certify Independent Upgrades

Abstract

Independently improved agents can form a worse team. We study when tests of single and paired replacements certify every allowed mixed-version deployment. Bounds on policy changes define an exactly realizable activation class. We derive its sharp pairwise remainder and prove that, for agents with equal exposure , the anchored predictor is minimax exactly when . Verified causal dependencies give smaller remainders and polynomial target bounds. Synthetic experiments compare different statistical information: on a 16-agent workflow, pairwise testing certifies three of three pools at 64 million episodes with a residual-coupling known-moment bound, and at 128 million with empirical Bernstein and common random numbers. Independently fitted 12–16-agent teams include compatible and incompatible upgrades; pilot-allocated exhaustive testing narrows some cost gaps. Multi-step local tasks reveal failures missed by uncorrected pairwise fits. Episode gates certify only the gate-averaged canary mixture; small gains require low gate rates and large evaluation budgets. Exact feasible intervals and attained-remainder sampling separate uncertainty caused by missing information from conservative statistical bounds. The results identify when low-order tests are informative, optimal, and cheaper than direct evaluation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multi-agent systems, policy upgrades, compatibility testing, statistical certification, causal dependencies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Cu22OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when tests of single- and pair-agent replacements certify that every allowed mixed old/new deployment of n independently upgraded cooperative agents is ε-compatible with the baseline. Given uniform per-history total-variation bounds did_i on each policy change over horizon HH (exposure pi=1−(1−di)Hp_i = 1-(1-d_i)^H), it characterizes exactly the realizable value tables (the "activation class", Thm. 1). It derives the sharp worst-case remainder of the anchored pairwise predictor (Thm. 2, ≤4∑pipjpk\le 4\sum p_ip_jp_k), and proves this predictor is minimax iff (1−p)m−2≥1/2(1-p)^{m-2}\ge 1/2 under equal exposures (Thm. 4), with a matching information-obstruction construction (Thm. 8). Two extra information sources tighten the bounds. Verified causal cones give local remainders and a polynomial McCormick target search (Thm. 5). Independent episode-level gates make exposure horizon-free but certify only the gate-averaged canary mixture (Thm. 6). Finite-sample certificates use empirical Bernstein bounds on coupled batches, or a known-moment bound valid under the residual-flag coupling (Thm. 7). Validation is entirely synthetic:

  • a 3-state, horizon-3 workflow (n = 8–16, 3 pools per cell);
  • independently fitted tabular teams in a shared-resource task (n = 12, 16; 36 pools);
  • a reservation-queue task with hidden incompatible triples (108 pools, exact means);
  • cost studies for gates.
Critical strengths
  1. Exact, sharp characterization of what low-order tests can and cannot certify: Thm. 1 gives an exactly realizable class rather than a loose envelope. Thm. 2's remainder is attained with both signs. Thm. 4 gives an if-and-only-if threshold for minimax optimality of the anchored predictor, with an explicit construction (App. O). Thm. 8 / App. C shows the ambiguity is an information limit that more samples at the same configurations cannot remove. This matters because it gives a principled reason to abstain, and separates "collect more samples" from "collect different information" (Sec. 8). I spot-checked the stated numbers (e.g. (0.98)14≈.754(0.98)^{14}\approx .754, B2≈1.60×10−2B_2\approx 1.60\times10^{-2} at m=16, p=.02, the threshold 1−2−1/14≈.04831-2^{-1/14}\approx .0483, the gate example ≈4.68×10−3\approx 4.68\times10^{-3} and −2.06×10−3-2.06\times10^{-3}, and the Table 3 entries), and they are consistent.
  2. Statistically careful certificates: Empirical Bernstein is applied to the full contrast row, so common-random-number correlation is accounted for (App. H). The known-moment bound is restricted to the coupling where it is valid. Pilot allocation uses discarded pilots and fresh holdouts (Prop. 14). Unadjusted maxima over methods or budgets are explicitly avoided (App. H, "What is and is not uniform"). LP bounds are verified with rational dual certificates (App. E). This matters because, as stated, the certificates can be trusted, which is rare in this kind of evaluation work.
Critical weaknesses
  1. The regime where the guarantees are both valid and useful is narrow, and the paper doesn't show it arises from realistic updates.
  • Small exposure is built in: The global certificate needs a uniform, sup-over-histories TV bound did_i that compounds with the horizon. In every experiment, small exposure comes from how the candidates are constructed: a per-round intervention probability in Sec. 6.1, and an explicit mixture with the old policy (mixture 6×10−36\times10^{-3}) in Sec. 6.2. For arbitrary fine-tuning, including the neural/LLM teams that motivate the paper, did_i can be close to 1 at some history (acknowledged in Sec. 8).
  • The advantage is costly and fragile: Even in the constructed regime, certification takes 16–128M episodes for n = 12–16 on 3-state, horizon-3 tasks. Raising exposure to p = .05 at n = 10 already reverses the advantage (Table 1). Gates cost roughly 1–17B observations and certify only the canary mixture, not the rollout (Sec. 6.4).
  • The abstract's claim to "identify when low-order tests are … cheaper than direct evaluation" is supported only where the exposure bound is supplied by design. That limits significance for the ICLR audience. They need to show one non-constructed source of small did_i, e.g. a verified bound on reachable histories (Sec. 4.1 / App. J.1) for a trained, non-mixture policy pair on a non-tabular task. Otherwise, rescope the claims explicitly to mixture- or gate-constructed updates.
  1. Weak empirical evidence that the procedure catches small-change incompatibilities under sampling.
  • No informative incompatible cases with sampled data: In Sec. 6.1 all pools are compatible. In Table 2, the incompatible teams appear only in the "larger update" regime, where every method abstains, so "0 false approvals" tells us nothing.
  • The informative case uses exact means: The hidden-triple case where pairwise fitting actually fails (Sec. 6.3) is evaluated with exact means and supplied causal cones.
  • Sampling looks very conservative: On the related small-margin split, Sec. 6.4 reports the local remainder plus statistical radius at a median 28.2× the triple margin with 32,768 rows per pair query. This suggests the sampled certificate may abstain broadly.
  • Few pools per cell. Table 1 cells have 3 pools on a coarse budget grid (16/64/128/256M), so the location of the crossover is imprecise.
  • Soundness is proven, but the claimed contribution is a usable decision procedure. Whether it approves compatible teams while rejecting small-change incompatible ones at feasible budgets is not shown.
Questions for authors
  1. Beyond explicit mixtures and gates, how would a practitioner obtain a valid did_i for a fine-tuned neural policy pair? Is there any task where the reachable-set restriction (Sec. 4.1) gives exposure small enough (e.g. p≤.05p\le .05 at n = 16) for a non-mixture update? A concrete example would substantially raise my rating.
  2. For the reservation-queue task (Sec. 6.3), what happens with sampled rather than exact means? At what budget does the local certificate approve the compatible pools and reject the incompatible ones, and how does direct EB compare at equal budget?
  3. Can you add incompatible teams in the small-update ("underused") regime of Sec. 6.2, where pairwise testing actually certifies, so the false-approval rate is measured where it matters?
  4. The direct comparators union-bound over K=2n−1K = 2^n-1 targets. How much of the crossover comes from this union bound rather than from the configuration count? Would a direct procedure that adaptively searches for the worst mask (e.g. with time-uniform confidence sequences, as you cite from Howard et al., 2021) close the gap?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Cu22 (1/3): exposure, rescoping and fine-tuned candidates (W1, Q1)

Thank you for the careful review and for checking our numbers. Below, budgets are in millions (M) of episodes, EB is empirical Bernstein, derived marks a number computed from the paper's formulas but not printed in it, and W2a–W2e are the five points of your W2 in order.

W1: small exposure is built in. We agree that in every sampled experiment, small exposure came from how we built the candidates. Without their mixture, the Sec. 6.2 candidates differ from the old policies with total variation .92–.998 (Appendix Q.1), which would make the global certificate vacuous. The abstract sentence you quoted is gone. The revised abstract, Sec. 1, Sec. 6 and Sec. 8 claim smaller budgets only in synthetic tasks where exposure is built in or verified after fine-tuning with a Kullback–Leibler (KL) penalty whose weight is tuned to the exposure target, and only against direct testing with equal or pilot-based allocation.

Q1: a source of small did_i that is not built in. Appendix Q.8 gives a partial answer. Each learner fine-tunes a small residual network (32 tanh units) on its old policy's logits with a KL penalty toward the old policy. We compute did_i exactly and use the smallest penalty weight that gives di≤.006d_i\le.006, the paper's mixture weight, which keeps exposure below .02 over H=3H=3 rounds (derived). Because every round–state pair in this task is reachable (Appendix Q.1), the restriction of Sec. 4.1 does not shrink did_i here, and the small exposure comes from the tuned penalty. Of the 24 compatible teams, pairwise EB certifies every underused one, from 16M episodes at n=12n=12 and from 64M at n=16n=16. Pilot EB certifies the n=12n=12 teams at 64M but none at n=16n=16 at 128M, equal-allocation direct EB certifies none, and every method abstains near capacity. If agents see only noisy continuous readings of the state, we bound did_i over all real-valued readings by interval arithmetic, and pairwise EB again certifies every underused team, at 64M for n=12n=12 (where pilot EB ties) and at 128M for n=16n=16.

The task is still our 3-state task, and the penalty weight must be tuned: at a fixed weight of .1, did_i reaches .061. Although the update is trained rather than built as a mixture, in the underused regime it can be rewritten as a mixture with weight at most .00602, because training raises the request probability at every round and state. A logit residual capped so that Lemma J.1 holds is valid for any network by construction, but here its single gains are only 2.72.7–3.8×10−53.8\times10^{-5}, about a hundredth of the KL-regularized gains (Appendix Q.8). Exact verification does not scale to long histories, large networks or language models, and we have no verified example there (Sec. 8). Valid sources of did_i are exact computation, mixtures, verified reachable sets and bounded logit residuals, while bounds averaged over visited states or maxima over sampled histories are not (Appendices J.3–J.4).

Costs and horizon. Budgets are large because small exposure also caps each single gain at pip_i (eq. 17 in Appendix A). In every pool of Table 1 and every underused pool of Table 2, the worst mix is a single upgrade (Appendices Q.4 and Q.6), and in the underused pools its gain is only about 3.7×10−33.7\times10^{-3}. At p=.05p=.05 the derived B2B_2 is about .052, three times the .016 at n=16n=16, p=.02p=.02, while Theorem 4 shows that the predictor is minimax-optimal for the equal-exposure version of that row, since (0.95)8≥1/2(0.95)^8\ge1/2. The reversal therefore most likely reflects the larger remainder, not a weak predictor. Exposure also compounds with the horizon: keeping p≤.02p\le.02 needs di≤6.7×10−3d_i\le6.7\times10^{-3} at H=3H=3 but 6.7×10−46.7\times10^{-4} at H=30H=30 (derived). Gates avoid this compounding but certify only the gated mixture, at a cost of about 1.09 to 17.46 billion observations (Sec. 6.4).

AuthorsOpenReview ↗

Response to Reviewer Cu22 (2/3): incompatible teams, the queue task and the small-margin split (W2, Q2, Q3)

W2a: no informative incompatible cases under sampling. We agree. Table 2's incompatible pools are exactly the larger-update pools, where every method abstains, and the revised caption now says so. Because Theorem 7 caps false approvals at α=.05\alpha=.05 whenever the exposure bounds hold, the informative quantities are power on compatible teams and the false approvals of the uncorrected pairwise fit.

Q3: incompatible teams at the exposure where pairwise testing certifies. The Sec. 6.2 underused teams cannot be incompatible given their own single and pair values, because their exact pairwise bound equals their smallest single gain, 3.7×10−33.7\times10^{-3}. We therefore planted incompatible teams in two games in the Sec. 6.1 setting (Appendix Q.9). The new Lemma Q.1 shows that ∣hij∣≤2pipj|h_{ij}|\le2p_ip_j and ∣gi∣≤pi|g_i|\le p_i, both attained, so near exposure .02 a pair with nonnegative single gains can fail only if those gains sum to less than about 8×10−48\times10^{-4}.

  • In the first game, two upgraded agents conflict when both intervene in the same episode, like two agents claiming one shared resource. Although every single and pair upgrade helps, 41 of 96 teams are incompatible, and no method approved any of them. These were not close calls: the exact pairwise bound is at most −7.9×10−3-7.9\times10^{-3}, and even the uncorrected fit is safe. Of the 55 compatible teams, pairwise EB certified 24, the earliest at 64M. The other 31 were not certified at any tested budget, and for 26 of them the remainder exceeds the margin, so no budget would suffice. Equal-allocation direct EB certified none, while pilot EB beat pairwise EB in one setting (n=12n=12, 3 of 6 pools at 128M).
  • In the second game, the true gain of every mix equals its exact pairwise lower bound (the worst case of Theorem 2), so only the statistics decide. Here 24 of 56 teams are incompatible by only 1.2×10−51.2\times10^{-5} to 2.5×10−42.5\times10^{-4}, through barely harmful single upgrades. No method approved any of them, although pairwise-EB lower bounds came within 5.5×10−55.5\times10^{-5} of zero. The uncorrected fit's lower bound exceeded the true gain in 112 of 168 pool–budget pairs, so it is not a valid bound. Compatible teams with margin at least 1.9×10−41.9\times10^{-4} were certified, from 256M at n=12n=12 and 512M at n=16n=16, but margins of order 10−510^{-5} were not resolved.

When we raise the Sec. 6.2 mixture weight toward the compatibility boundary (Appendix Q.6), the exact pairwise bound is negative from .02 on, and direct EB certifies instead, up to .2.

Q2 and W2b: the queue task from sampled episodes (Appendix Q.5). We sampled all 108 Sec. 6.3 pools at budgets up to 1,024M. No sound method approved any of the 54 incompatible pools, and no coverage failure occurred. Because the exact local bound is already negative on every incompatible pool, this experiment measures power on the compatible pools, and Appendix Q.7 tests teams near the compatibility boundary. Local pairwise testing certified all compatible pools at 64M, 256M and 1,024M for n=12n=12, 24 and 48. We ran direct testing only at n=12n=12, H=3H=3 (six compatible pools), where direct EB certified 5 of 6 pools at 1,024M and pilot EB all 6 at 256M. At n=24n=24, direct testing would need about 1.6×1091.6\times10^9 episodes just to bring its EB range term below the return range. The sampled uncorrected fit falsely approved more incompatible pools as the budget grew, reaching all 18 at n=12n=12 from 256M on. Within-cone exhaustive testing, which uses the same causal cones but tests every subset within each cone, certified one budget step earlier, so it is the better design for three-client cones (Sec. 6.3 and Sec. 8). The cones remain supplied information that the evaluator must verify.

Appendix Q.7 sweeps η\eta, the hidden triple's gain in units of its scale, with the boundary at η=0\eta=0 and the exact local threshold near η=3\eta=3. No sound method approved an incompatible team, while the uncorrected fit approved teams about 2×10−52\times10^{-5} below the boundary from 64M on. Local pairwise testing abstained on compatible teams with η≤2\eta\le2, which within-cone testing certified.

W2c: the 28.2× statistic. The small-margin split is a separate one-step game with 3, 4 or 6 agents, with η∈{±.1,±.3,±.9}\eta\in\lbrace\pm.1,\pm.3,\pm.9\rbrace as above. Even the exact local pairwise bound is negative on its compatible half (Sec. 6.4). The remainder alone is 4/∣η∣4/|\eta| times the margin (median 13.3), and the statistical radius adds a median 13.5 (Appendix Q.3). Both parts matter: more samples shrink only the radius, which is itself conservative, since EB coverage-failure rates were 3.00×10−43.00\times10^{-4} to 1.80×10−31.80\times10^{-3} against a nominal .05 (Sec. 6.4).

AuthorsOpenReview ↗

Response to Reviewer Cu22 (3/3): more pools, union bound and adaptive testing (W2d, Q4)

W2d: three pools per cell. We agree that three pools give an ordering at the tested budgets, not a crossover location. The derived 95% Clopper–Pearson interval for 3 of 3 approvals starts at .29. Appendix Q.10 adds 20 pools to each Table 1 cell and uses a finer budget grid. All pools are compatible, so these runs measure power, and no coverage failure occurred. Pairwise EB certified all 23 pools at 6M for n=8n=8 and at 32M for n=12n=12, while pilot EB certified 21 of 23 at n=12n=12 with 64M. At n=16n=16, pairwise EB certified 20 at 128M and all 23 at 192M, while neither direct design certified any pool up to 512M (pilot) or 1,024M (equal allocation). At n=10n=10, p=.05p=.05, direct testing wins: pilot EB certified all 23 at 8M, but pairwise EB only 18 at 512M. Table 1's orderings therefore hold.

Q4: union bound or configuration count? At n=16n=16 the gap comes mainly from the configuration count. Pairwise and direct certificates pay the same union bound over all targets (Theorem 7). In the cost envelopes of Appendix N, the union bound multiplies every design's cost by the same factor, from 2.8 at n=8n=8 to 4.7 at n=16n=16, so it largely cancels between designs (Appendix Q.2). We also recomputed the certificates without the union bound (K=1K=1) on the same sampled rows as Table 1, regenerated from the paper's seeds (Appendix Q.4). These bounds are not valid for the whole family. Equal-allocation direct EB then certifies n=12n=12 at 128M but still none of the n=16n=16 pools up to 256M, where each of the 65,536 mixes gets only 976–3,906 rows. An oracle that spends the whole budget on the true worst mix, always a single upgrade, certifies at every budget. Direct testing is limited because it must spread its budget over all mixes without knowing which is worst.

Q4, second part: adaptive worst-mix search (Appendix Q.4). We implemented a valid adaptive direct method. Its time-uniform error control comes from a union bound over mixes and geometrically spaced checks rather than from the confidence sequences of Howard et al. (2021). It samples all mixes not yet certified with shared random numbers, checks EB bounds after 64⋅2j64\cdot2^j rows with error α2−(j+1)/(2n−1)\alpha2^{-(j+1)}/(2^n-1) at check jj, drops certified mixes, and charges every episode. At n=10n=10, p=.05p=.05, it certifies at 7.7–8.7M, about the same as pilot EB (8M). It needs 7.4–8.1M at n=8n=8, more than pairwise EB (6M), and 78–87M at n=12n=12, 2.4–2.7 times pairwise EB (32M); both baselines come from the finer grid of Appendix Q.10. At n=16n=16 it certifies at 1.04–1.15 billion episodes, above the largest budgets at which we ran the fixed direct designs and eight to nine times pairwise EB (128M, Table 1). The comparison is not exactly matched, because the fixed designs are read off budget grids while the adaptive method pays a logarithmic price for being valid at every check. Sec. 6.1 and Sec. 8 report this, and Sec. 7 now cites work on good-arm identification and confidence sequences.

W2e: a usable decision procedure. Across Appendix Q, pairwise testing approved compatible teams at budgets from 6M to 1,024M episodes, and no sound method approved an incompatible team. Pairwise testing abstained on compatible teams whose margin is of order 10−510^{-5} or below the remainder (Appendix Q.9). Sec. 8 states that we characterize power only for these settings.

Reviewer_hsPVOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies multi-agent systems, starting from the observation that improving individual AI agents does not necessarily improve the overall system's performance. Specifically, the author(s) propose a framework to test if upgrades to the agents do not hurt overall team performance. Instead of relying on a brute-force approach, the author(s)' algorithm test (i) the original team, (ii) each upgraded agent individually and (iii) pairs of upgraded agents. Although this framework is much more efficient that brute-force, it does not take into account interactions between three or more agents. The author(s) provide theoretical analysis, showing when testing pairs of upgraded agents is beneficial. The author(s) further validate the proposed framework on synthetic data consisting of 8-48 agents.

Critical strengths

I think that the problem of understanding multi-agent systems is relevant. I found that the paper is overall easy to follow. Since this work falls outside of my field of expertise, I cannot comment on the relevance and novelty of the proposed approach, or the theoretical framework.

Critical weaknesses

It seems to me that the experimental framework is rather weak: the author(s) only consider synthetic data and simulated interactions.

Questions for authors

Can you please comment on (1) the novelty of your approach, (2) the novelty and technical challenges of the theoretical contribution? In your work you only consider synthetic data. Can you provide evidence that the proposed framework relevant for real-world applications?

Given that this work follows outside of my field of expertise, I am willing to review my score depending on the rebuttal, as well as further engagement with the other reviewers.

1 comment
AuthorsOpenReview ↗

Response to Reviewer hsPV: higher-order interactions, novelty, theory and real-world relevance

Thank you for the review and for being open to updating your score. Below, a mix is a team in which some agents run their new versions and the rest their old ones, and an upgrade's exposure is its chance per episode to act differently from the old version.

Interactions among three or more agents are bounded, not ignored. The pairwise prediction of a mix's return leaves these interactions out, but our approve-or-abstain decision, the certificate, subtracts a worst-case bound on everything the prediction omits (Theorem 2) and abstains when that bound is too large (Theorem 7). In the reservation task of Section 6.3, 54 of 108 teams hide a conflict among three agents. With exact expected returns, plain pairwise prediction approves all 54 conflicting teams, while our certificate, which also uses verified knowledge of which agents affect which rewards (Theorem 5), approves all 54 good teams and none of the conflicting ones. With enough sampled episodes (new Appendix Q.5), the certificate again approves exactly the good teams, while plain pairwise prediction falsely approves more conflicting teams as episodes accumulate.

Pairwise testing is not always cheaper, and the paper aims to say when each approach is preferable. Testing every mix directly wins at 10 agents with exposure .05 (Table 1). At 12 agents, direct testing guided by a small pilot sample ties with pairwise testing at the episode budgets of Table 1, but not on a finer grid of budgets (Appendix Q.10).

(1) Novelty of the approach. Testing singles and pairs resembles factorial designs (Box et al., 2005) and tt-way testing, which covers every combination of settings of any tt parameters (Kuhn et al., 2004). Work on configurable software measures features singly and in pairs and handles higher-order interactions heuristically, without a worst-case bound (Siegmund et al., ICSE 2012). Separately correct components are known to conflict (Calder et al., 2003; Sculley et al., NeurIPS 2015), and update-compatibility work studies regressions when one model is replaced (Bansal et al., 2019). To our knowledge, three things are new. From tests of only the original team, each single upgrade and each pair, plus verified exposure bounds, we certify every allowed mix with a stated error probability. We separate abstention caused by missing information from abstention caused by statistical noise (Section 6.4). We also compare the costs and failure modes of three sources of extra information: exposure bounds, verified causal structure and random episode-level switches (gates).

(2) Novelty and challenges of the theory. Without extra information, an upgraded agent can change everything that happens later in an episode, so certifying every mix would require testing essentially all of them. We assume instead a verified bound on each upgrade's exposure. Theorem 1 shows that the possible tables of average returns are then exactly the averages over which agents get a chance to act differently, where an agent without that chance acts as its old version would (Figure 1). Theorem 2 then gives the exact worst-case error of the pairwise prediction. The hard parts were three proofs: that every such table comes from an actual multi-agent process (Appendix A), that the error bound is attained in both directions (Appendix B), and, for equal exposures, the exact condition under which no other way of combining the same tests has a smaller worst-case error (Theorem 4). For three agents, Theorem 8 constructs two teams that agree on every smaller mix, so when only team returns are observed, more samples of those mixes, even chosen adaptively, cannot tell them apart. The certificates use standard concentration inequalities, and Theorem 3 specializes classical results on optimal estimation from linear measurements.

(3) Synthetic data and real-world relevance. We agree that synthetic data is a real limitation (Section 8). Counting false approvals needs the exact value of every allowed mix, which our synthetic tasks provide (65,536 mixes for 16 agents). For a real team these values could only be estimated, and we have not run such a study. The deployment pattern itself is common, since components ship on separate schedules, customers enable different subsets, and platforms run many concurrent changes (Tang et al., KDD 2010; Kohavi et al., 2020). Appendix Q.8 adds a recipe for small neural policies: fine-tune each agent with a penalty toward its old policy, and tune the penalty until the verified exposure meets a chosen limit. In our synthetic shared-resource task (Section 6.2), pairwise testing then certified every team whose shared resource was lightly used. We have not shown that our method scales to language-model agents.

Reviewer_ZXAKOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper is not human-readable. It strings together an endless array of jargon and technical terms into indecipherable sentences, without providing any definitions or explanations. Take the very first three paragraphs of the Introduction as an example: What are behavior change, cooperative process, exposure, return, low-order values, untested interaction, target, value table, recoverable, activation representation, matching construction, anchored pairwise predictor, and small-change extrapolation? The paper does not provide a single reference that introduces or contextualizes the problem being studied until the penultimate section, apart from several technical citations about optimal recovery and concentration after Section 2. Therefore, I am unable to provide a meaningful review of this paper, and per reviewer guidelines, I do not intend to use an LLM, as I believe authors already have access to such tools.

Critical strengths

Cannot identify

Critical weaknesses

Cannot identify

Questions for authors

Cannot identify

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZXAK: definitions and the revised introduction

Thank you for the specific feedback. You are right that the abstract and introduction used the terms you list without defining them and that Sections 1–6 cited no work putting the problem in context. The revised PDF rewrites the abstract and Section 1.

The problem in plain words. Each agent in a team gets its own new version, and each new version can be better on its own, yet some mix of old and new versions can make the team worse. We ask when tests of the original team, each single upgrade and each pair suffice to guarantee that no allowed mix is worse than the original by more than a tolerance.

Where your terms are now defined. Eight of the thirteen terms are defined on pages 1–2.

  • On page 1, exposure is the probability that an upgrade gets a chance to act differently from the old version during an episode, and return is the team's score in an episode.
  • Also on page 1, low-order values, now called low-order tests, are the results of testing the original team, each single upgrade and each pair, and a target is an allowed mix whose effect must be guaranteed.
  • On page 2, a value table lists the expected returns of all mixes. The anchored pairwise predictor predicts a mix's expected return from the original team's return, each single upgrade's gain and, for each pair, how much its joint gain differs from the sum of its two single gains.
  • Also on page 2, a matching construction consists of two teams that give identical results on all low-order tests. With all agents upgraded, one team scores exactly the predictor's worst-case error (Theorem 2) above the prediction and the other the same amount below it.
  • Section 2 opens on page 2 by defining a cooperative process as an episodic environment in which several agents act and share one bounded team score.

The main text no longer uses the other five terms. Behavior change is replaced by did_i, a verified bound on how differently the new version can act in any situation (equation 2, page 3). Activation representation is replaced by the plain statement that only agents that got a chance to act differently matter (page 2 and Figure 1). Untested interaction, recoverable and small-change extrapolation are replaced by direct descriptions in Section 1.

Other changes for readability. Section 1 now has a paragraph with citations on related problems (update compatibility, teamwork with new partners, factorial and tt-way testing, feature interactions) and a list of contributions (page 2), and a notation box summarizes the symbols (page 3). Sections 3 and 4 now also define symbols where they are first used, for example mm, the number of upgraded agents in a target. Please tell us if anything is still unclear.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: what we changed, and what we do and do not claim

We thank the three reviewers and accept their main points: the paper was hard to read (ZXAK), its experiments are synthetic (hsPV, Cu22) with small exposure built in (Cu22), and its evidence from sampled episodes was thin (Cu22). An upgrade's exposure is the probability that it gets a chance to act differently from the old version during an episode. We have uploaded a revised PDF in which Appendix J.4 and Appendix Q are new and all other numbering is unchanged.

The abstract and Section 1 are rewritten in plain language, and the main text now either defines each term Reviewer ZXAK listed or no longer uses it. The abstract and Sections 1, 6 and 8 now claim that testing singles and pairs saves episodes only in synthetic tasks where exposure is built in or verified after fine-tuning with a per-agent tuned penalty toward the old policy. Even there, the only comparison is with testing every mix of old and new versions directly, with episodes split equally or by a small pilot sample. Appendix Q adds experiments on each of Reviewer Cu22's four questions, and our answer to the first is partial.

We do not claim results for real language-model or robotic teams. Nor do we claim that unconstrained or large-scale fine-tuning yields small exposure, or that testing singles and pairs beats every way of testing all mixes directly. Gates, which switch each upgrade on in only a random fraction of episodes, certify only that randomized deployment.

AuthorsOpenReview ↗

Summary of revisions for the AC and reviewers

We thank the reviewers again. The revised PDF rewrites the abstract and Section 1 in plain language and adds a notation box and self-contained table captions. It also restricts the cost claims to synthetic tasks where exposure is built in or, in Appendix Q.8, verified after penalized fine-tuning. In Appendix Q, fine-tuned teams with verified exposure are certified when their shared resource is underused (Q.8), and Table 1's orderings hold with 23 teams per setting (Q.10). No sound method approved an incompatible team in the sampled queue task (Q.5) or in two games at small exposure (Q.9).

Section 8 states the remaining limits. We have no real language-model or robotic evaluation, we verified small exposure only for small memoryless policies, and gates certify only the randomized deployment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.