acceptodds
Under review as a conference paper at ICLR 2027

A Conservation Law for Learning in Strategic Systems

Abstract

Standard exploration diagnostics measure how much a policy moves, but not whether that movement remains informative after the data-generating process responds. We introduce strategic observability to measure how strongly policy motion separates latent environments after this response, and define strategic capacity as motion times squared observability. For binary environments, mutual information is bounded above by expected capacity and below by a truncated almost-sure capacity floor. Under local Fisher geometry, the quadratic exploration cost decomposes orthogonally into identifiable and null components. Their relative weights are the squared cosine and sine of the angle between policy motion and the identifiable subspace; the identifiable component controls learning up to the conditioning of the identification operator. Local characterisation results further select Fisher separation energy as the learning currency and the symmetric Fisher quadratic form as the cost. The finite-step quantities used in practice are measurable surrogates for, rather than exact copies of, these geometric objects. Empirically, six policies with exactly equal budget, average separation energy, switching, exposure, and action entropy separate only through minimum pairwise separation: the two with zero minimum perform at chance, whereas the remaining four achieve error below . A language-model intervention likewise shows that greater motion along an uninformative direction can yield negligible separation.

Primary area:
reinforcement learning
Keywords:
Multi-agent learning, conservation law, strategic systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DEMDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that standard exploration diagnostics (action entropy, switching, drift) measure policy motion, not whether that motion stays informative once data-generating agents respond.

It factorises exploration into a KL-drift cost Δ_t and a strategic observability O_t, defines strategic capacity C = Σ O_t^² Δ_t, and splits the information budget exactly as SIB = C + W, with efficiency η = C/SIB.

The Theory: a binary-hypothesis information bound: a.) Theorem 1 with a truncated capacity floor, local Fisher geometry splitting motion into identifiable and null components b.) Theorem 2. , Chentsov-type characterisations, a frontier sandwich, and Fano corollaries; a commons model yields a parameter-free log-precision prediction (½ log N) . c.) Experiments: a budget-matched mirage where six policies identical in every motion diagnostic separate only through minimum pairwise separation (Tab. 1), an interventional test with a real LLM responder where blind outspends targeted by 44% of motion for 0.4% of the separation (Tab. 2), a recommender collapse experiment, and a weak-signal replication using human preference data.

Critical strengths

The paper identifies a genuine gap and demonstrates it with a sharp instrument!

  Equal-budget policies spanning the whole error range while identical in entropy, switching, exposure, and average separation is exactly the kind of matched counterexample that makes a diagnostic claim hard to dismiss, and the interventional LLM test goes beyond author-designed response maps in at least one system.

Very direxct and explanibale elements and real limitations comments.

The paper flags that ĈT is an unnormalised surrogate because C_up is unidentified, admits Assumption 2 fails globally in its own headline counterexample, reports the degraded fit in the weak-signal replication with intervals, and includes an AI-use statement.

Critical weaknesses

2 critical comments

The central decomposition is an accounting identity: O_t is defined to make SIB = C + W hold, so the only measured quantity is accumulated squared separation; C_up is never identified, leaving "capacity" and "efficiency" η as unnormalised surrogates. No decision problem shows that factorising SIB changes outcomes versus accumulated separation, EIG, IDS, or a Hellinger/Chernoff tracker; since the mirage turns on the minimum pairwise separation, which the framework does not foreground, theory and ad-hoc summary are confounded.

Many empirics largely confirm built-in structure: author-designed response maps, a known parametric estimator, a by-construction counterexample, and a lone external test that fits worse without analysis. The headline R² = 0.92 vs 0.07 leaves unit counts, held-out status, and the ĈT > 0.01 threshold rule undiscussed, with no error treatment for the regression (unlike in Table 1). Assumptions 2, 5, 6 and the ess-inf floor are never checked empirically.

Questions for authors
  1. In any decision problem wheter it is policy selection, stopping, or budget allocation across systems, does the capacity C_T outperform accumulated squared separation ĈT, an EIG/BALD-style criterion, IDS, or a worst-pair Hellinger tracker? If not, can you explain waht the factorisation add?

  2. Can C_up be estimated or bounded in even one system, and how do the reported rankings and R² change under plausible misspecification of it?

  3. What formally distinguishes a "strategic" response from a generic policy-dependent kernel ?

  4. For the R² = 0.92 vs 0.07 results: how many units, is evaluation held out, and how sensitive is the fit to the ĈT > 0.01 threshold used to select these 29 policies? Could report and comment the same regression without thresholding.

  5. Could all or some of the sssumptions 2, 5 and 6 be checked empirically, and how non-vacuous are the constants? In particular, report empirical lower quantiles of S_T and the conditioning of J_t in the RLHF and recommender systems, where geometric freshness decay should stress the reverse-Pinsker regularity.

  6. The mirage is decided by minimum pairwise separation while the theory centres on summed separation: is there a version of the theory that would predict the binding pair, or is this an unexplained regularity of the construction?

  7. in a completely different fields and within very different contexs sinfotaxis (for olfactory searches) and infotaxis derived strategies for multi armed bandit have demonstrated efficient policy and strategy decision. Can you relate your work to these ?

2 comments

Thank you for comments. Please see the general rebuttal and specific responses below.

1. Does the factorization improve a decision? At the largest threshold we tested, yes. Appendix W compares four policies in the same synthetic system. Their accumulated hardest-pair squared TV and Hellinger summaries are nearly identical. The spreads are 1.7×10−51.7\times10^{-5} and 1.3×10−51.3\times10^{-5}, respectively. Their full-horizon separation per unit motion differs by a factor of 1.5. At the first step reaching the threshold B=8B=8, the efficiency ranking matched the observed error ranking. The policy ranked first by efficiency, ε-greedy, had 4.2% observed error, compared with 6.5% for the highest-error policy at that threshold. At B=2B=2, the posterior-guided policy, which uses expected information gain, had the lowest observed error. Appendix W reports all four thresholds and defines the evaluation at the first crossing, which can include overshoot. The Hellinger comparison uses accumulated summaries rather than a separate adaptive Hellinger policy.

Across systems, both preregistered allocation experiments missed their success criteria. In the first, allocation by separation per unit cost beat allocation by accumulated separation at three budgets and lost at the fourth. The largest difference occurred near a discrete jump in the error curve. In the second design, which reduced those jumps, equal splitting had the lowest observed mean error at all four budgets. At B=40B=40, mean errors were 0.0760 for equal splitting, 0.0782 for accumulated separation, 0.0845 for separation per unit cost, and 0.0770 for the model-weighted oracle.

2. Can CupC_{\rm up} be estimated or bounded? We have no empirical estimate of CupC_{\rm up} for the reported systems. The fitted predictor is unnormalized separation energy. Multiplication by a common positive constant within one system would preserve its rank order and the R2R^2 of the same intercept-including regression on the same units. An absolute cutoff would need to be rescaled as well.

3. What makes the response strategic? Our model specifies a response map from the learner’s announced policy or continuation and the latent environment to the state entering the observation kernel. Marginalizing that state gives a policy-dependent kernel. A generic policy-dependent kernel can represent the same observable law. The response map matters when we ask what the responder would do if the learner announced a different policy. Standard bandit observation laws can also differ greatly in how informative their actions are.

4. How many units and how sensitive is the fit? We fit the regression to one outcome per policy in the synthetic RLHF sweep. With all 32 policies, R2R^2 is 0.874 for separation energy and 0.259 for the motion budget. The cutoff of 0.01 retains 29 policies and yields 0.917 and 0.068, respectively. Every cutoff from 10−610^{-6} to 0.05 retains the same 29. All three omitted policies have zero separation energy and error 0.5. The revised main text leads with the all-policy fit.

5. Do the assumptions hold and are the constants informative? In the synthetic Gaussian model, the shared-variance formula gives KL/TV2\mathrm{KL}/\mathrm{TV}^2 between π\pi and 4.2912 on the specified range. As the mean gap decays, the ratio approaches π\pi. The global no-static-signature condition fails in the matched example and in the LLM intervention. For each of the two blind policies in the matched example, the relevant response laws agree on its unresolved pair. Across 2,400 synthetic runs, the smallest observed mean-pair STS_T was 0.7974. Minimum-pair separation was zero in one third of runs and had median 2.3923 in the remainder. The observed minimum describes these runs. The theorem’s essential-infimum condition concerns all histories up to a null set.

6. Why does the minimum pair matter? Each blind policy leaves a pair that produces exactly the same transcript law. Conditional on that pair and with equal prior odds, the best possible error is 50%. The simulated errors were 50.7% and 50.3%. The other pairs can contribute plenty of separation to the average without helping the learner tell these two apart.

7. How does this relate to infotaxis and IDS? Infotaxis chooses moves by expected information gain about a source. IDS balances information about the optimal action against expected regret. In our setting, either approach needs a predictive law that accounts for how the responder reacts to each candidate policy. The local analysis characterizes directions in which that law supplies little or no discrimination per unit motion. Infotaxis · Information-directed sampling

The revision addresses all your concerns and includes two new results.

Q1. Policy selection and budget allocation. Within a system, accounting for cost changes the policy ranking. The four informative policies share the same realised separation sequence on their hardest pair. Their accumulated total-variation and Hellinger scores agree to within a relative spread of 7×10−67 \times 10^{-6}, leaving them effectively tied on both measures. At the largest budget, the efficiency criterion selects the policy with the lowest observed error (4.2%, compared with 6.5% for the highest-error policy). The Spearman correlation between efficiency and error is −1.00-1.00.

Q2. Local constants. In the local regime, the averaged cost is 12v⊤Gv\tfrac12 v^\top Gv. For two nonnegative costs, their maximum lies between their mean and twice their mean. It follows that CupC_{\mathrm{up}} is the largest eigenvalue of GG, while clowc_{\mathrm{low}} is half its smallest nonzero eigenvalue. Their ratio is therefore twice the condition number on the nonzero eigenspace. The numerical tightness ratios are 0.989 and 1.000, respectively. Applying Weyl’s inequality to a 120-point ε\varepsilon-net certifies the uniform bound Cup≤8.2417C_{\mathrm{up}} \le 8.2417. A denser 7,132-point grid gives a reference value of 8.1006. We also show that the direct plug-in estimate approaches the constant from below and therefore cannot certify an upper bound.

Q3. Bandit comparison. Section 2 now explains the distinction using the identity-response bandit setting. With an identity response map, policy changes are transmitted directly, so the strategic response cannot erase separation.

Q4. Full-sample results. The main text now reports the fits across all 32 policies first. The R2R^2 values are 0.874 for separation energy and 0.259 for the budget. Both describe in-sample variation across policies.

Q5. Empirical constant and decay. The empirical constant is 4.2912, at the upper endpoint of the analytically derived range. As decay increases, the ratio moves toward π\pi, making the assumption easier to satisfy in this setting.

Q6. Zero capacity. The revised result gives an exact equality. At zero capacity, the two transcript laws are identical, so the transcript carries zero mutual information about the environment.

Q7. Infotaxis. We explain how infotaxis selects actions by expected information gain in the identity-response setting. We also report that it outperforms our efficiency criterion at the smallest budget tested.

If the revisions resolve your concerns, please reconsider your rating.

Reviewer_iERnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper focuses on evaluation of exploration when the environments responds to the learners' policy. The authors introduce strategic observability, the cross-environment separation produced per unit of policy motion, and strategic capacity, the motion cost weighted by squared observability. The first main result shows that, for two environments, mutual information is controlled (from above) by expected capacity, and (from below) by the truncated capacity. The second shows that in locally smooth environments, the quadrati part of a environment-averaged exploration cost can be decomposed in two orthogonal components, corresponding to the different/identical reaction of the environment.

Critical strengths

The authors cleanly demonstrate that motion-based exploration diagnostics in strategic systems is not sufficient to evaluate exploration.

Moreover, they propose a formal decomposition of exploration into motion and strategic observability, with (asymmetric) information-theoretic bounds for binary environments, and a local geometric interpretation.

Some experiments across different frameworks seem to confirm their findings.

Critical weaknesses

Inconsistencies are scattered around in the paper (non-exhaustive list):

  • Theorem 1 is stated for the accumulated squared endpoint separation, under Assumptions 1, 2, 6, but the proof, in appendix D, establishes the capacitary version, relies on Assumption 5, and obtains, at line 820-821, the non-universal constant C=κCup/clowC=\kappa C_\text{up}/c_\text{low}. It is unclear which statement is proved, and under which assumptions.
  • Assumption 5 states in line 729 that the upper constant CupC_\text{up} does not follow from Assumptions 3, 4. But Theorem 30 then gives Cup=(GxLcont/(1−ρ))2C_\text{up} = (G_x L_\text{cont}/(1−\rho))^2, that is CupC_\text{up} depends exactly on the constants appearing in Assumptions 3 and 4.
  • In Assumption 6, in lines 742-743, it is stated that uniformly bounded likelihood ratios do not suffice for reverse Pinsker. But, Remark 13, in lines 1024-1025, says that reverse Pinsker is available whenever likelihood ratios are uniformly bounded.

Assumptions are collected in Appendix B. it is unclear under which assumptions each result hold. Sometimes, for a result, assumptions are hidden in the text preceding the statement, sometimes in the proof in appendix, sometimes no assumption is mentioned. Moreover there is little to no discussion about the validity of the assumptions in the application domains invoked by the authors.

Theorem 1 is restricted to binary environments. The non-binary case is discussed in a short remark (Remark 21), and only one-sided bounds are given (Theorem 35)

Theorem 2 is an immediate consequence of orthogonal projection, and its proof is straightforward. It should be better presented as a lemma, or remark.

The experiment section has poor description of the experimental setups, of the protocols, and of their outcomes. Moreover in the experiment in Section 4.2 ``Assumption 2 fails globally'' (line 371).

Literature review should be improved, at least in the direction of interactive decision-making, incomplete learning

The paper needs careful and substantial rewriting because it is difficult to read as it is. There are several and critical issues (non-exhaustive list):

  1. The paper has not been written in a clear, linear, structured style that would help the reader to understand the content.
  2. Several symbols are used before being defined (Π\Pi in line 083, BRtBR_t in line 137, etc), or never formally defined (II in line 096, Ft−1\mathcal{F}_{t-1} in line 158). Sometimes they are left implicit and to be guessed by the reader (KL in line 140, TV in line 158, etc). None appear in the notation table of Appendix A.
  3. Several draft-like remarks, such as: * "An earlier version of this appendix..." in line 1381, * "...which is why we have renamed it." in line 1758 * "...the discarded per-step construction whose constant was 1/C_up." in line 1784 * “the asymmetry runs the opposite way from the one previously stated” (end of R.2) * the whole caption of Table 8 They refer to earlier version of the paper (I guess), and are meaningless in the present paper.
  4. Other artifacts (non-exhaustive list). * A broken reference in line 298. * The reference to T.1 in line 1174 is not correct * The sentence in line 1596 is incomplete. * Appendix T contains two almost equal paragraphs (lines 2015-2018 and 2043-2046). In line 2041 there is a reference to Appendix T... inside Appendix T. * Some sections (M.6, M.17) are empty. * There is a "Figure 4" caption on top of Figure 5 (page 20). * Often script filenames are cited inline to provide evidence.
  5. While relevant parts of the paper are either not discussed, or hidden in the appendices (see below about the assumptions), the main body contains uninformative sentences like the one in lines 187-188 (which is even repeated in lines 225-226!)
  6. Several results are first proved, and (sometimes much) later are stated. For example, Theorem 30 is proved in lines 965-980, and stated in lines 1740-1746.
Questions for authors

Could it be possible to include a discussion about the assumptions, beyond their technical validity?

Could it be possible to have a more through discussion of the non-binary case of Theorem 1?

Could you comment about Theorem 2? Could you quantify the gap between strategic observability and cos⁡2θt\cos^2\theta_t?

2 comments

Thank you for your review. Please see the general rebuttal and specific responses below.

Theorem 1 and the assumptions. The revised proof establishes the upper bound directly in terms of accumulated squared endpoint separation STS_T, with Assumptions 1, 2 and 6. The conversion to normalized capacity uses the two-sided drift comparison in Assumption 5, and belongs in the separate capacity corollary. The proof of the upper bound conditions on the observation history and uses the current posterior in the predictive mixture. Convexity gives a conditional increment of at most (κ/2)St2(\kappa/2)S_t^2. The theorem retains the stated constant κ\kappa. We treat CupC_{\rm up} as an assumption for the capacity comparison. A bounded likelihood ratio alone does not give a uniform quadratic reverse-Pinsker constant for arbitrary pairs of distributions.

Beyond two environments. The two-sided binary theorem is stated for two hypotheses. With several hypotheses, the paper gives an information upper bound using average pairwise separation and an identification-error lower bound using a hard pair. The multiclass proof now applies a separate chain rule to directed transcript KL. Corollary 34 optimizes the fixed-pair bound over expected pairwise capacities. Under the uniform prior, a fixed indistinguishable pair gives conditional chance error on that pair and therefore a lower bound of 1/M1/M on overall Bayes error. This explains how the average can look healthy while one pair remains unresolved.

The geometric result and its scope. We agree that the orthogonal splitting follows directly once the metric and subspaces are in place. We have revised the presentation to locate the mathematical content in the quadratic form and the identification operator. The upper comparison follows from the cost metric. A quantitative lower comparison needs a positive lower singular-value bound on the identifiable subspace. The size of the gap therefore depends on the channel, and it can grow as that lower singular value approaches zero.

Presentation and experiments. The revision adds an assumption-to-result table, expands the notation and protocol descriptions, and discusses interactive decision-making and incomplete learning. It now explains where the global no-static-signature assumption fails and why the unresolved pair in the matched example can still be analyzed directly.

Thank you for raising the question of proof soundness. We have addressed all issues.

  1. Theorem 1. The proof of the upper bound now uses the posterior-weighted predictive mixture required by the mutual-information chain rule. The stated constant is unchanged.
  2. Multiclass result. We now bound the directed divergence between transcript laws using its own chain-rule decomposition.
  3. Corollary 34. We replaced the expected pathwise minimum with the minimum of the expected pairwise capacities. The original substitution yielded a stronger bound than the proof established.

We also added the shared-baseline-kernel assumption to the canonicality theorem and the assumption table, since the expansion depends on it at leading order. The uncertainty corollary now takes the expectation required to bound the scalar Bayes error. We addressed the missing continuity requirement in the collapse theorem and supplied proofs for two propositions. For one of those propositions, endogenous kernel separability, we also corrected the pair of kernels in the statement. The original pair made the policy-mediated term identically zero. Finally, we made the lower-bound constant explicit as (1−e−1)/4(1-e^{-1})/4.

We would welcome any remaining concerns and will address them before the deadline. If the revisions resolve your soundness concerns, we ask that you reconsider your rating.

Reviewer_4PJgOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Note: While I have experience with empirical research on exploration in reinforcement learning, I have limited experience with theoretical research in this area. I would appreciate it if the AC could take this into account when evaluating my review.

This paper studies exploration in strategic systems where the environment adapts to the learner's policy. The authors argue that conventional exploration metrics, such as policy drift and action entropy, fail to distinguish informative from uninformative exploration. They introduce strategic observability, which measures how effectively policy changes distinguish latent environments, and propose a conservation law that decomposes exploration effort into strategic capacity (signal) and waste. They establish information-theoretic bounds relating capacity to mutual information and derive a geometric interpretation using Fisher information. Through synthetic experiments and an LLM-based intervention, they demonstrate that statistical separation better predicts identification accuracy than conventional exploration metrics, even when exploration budgets are matched.

Critical strengths
  1. Interesting and well-motivated problem. Distinguishing informative from uninformative exploration is an important perspective for learning in strategic systems, where the environment adapts to the learner's policy.

  2. Intuitive measurement framework. Separating exploration into signal and waste provides a useful way to reason about why greater policy variation does not necessarily lead to better learning.

  3. Well-controlled synthetic experiments. The budget-matched experiments demonstrate that policies with identical exploration statistics can have substantially different identification accuracy, supporting the importance of directional observability.

Critical weaknesses
  1. Limited theoretical novelty. The conservation law is algebraic by construction, while strategic capacity essentially reduces to accumulated squared statistical separation. Its contribution beyond established information-theoretic and identifiability results remains unclear.

  2. Limited practicality of the proposed measurement. Computing strategic capacity requires knowing how different latent environments respond to policy changes, which is generally unavailable in practice. The proposed estimator relies on a known hypothesis class and a specific parametric response model, limiting its applicability.

  3. Insufficient empirical validation. Experiments mainly show that statistical separation predicts identification accuracy, rather than directly validating the capacity–waste decomposition. Most rely on synthetic environments and unnormalized separation energy, with limited LLM evaluation and comparisons against existing informativeness metrics.

Questions for authors

N/A

2 comments

Thank you for your comments. Please see our general rebuttal and specific response below.

1. What does the theory add beyond accumulated separation? The new use of the decomposition is to diagnose where the motion went. In §3.2, we derive the environment-averaged Fisher metric for the cost of a local policy change and the operator that maps that change to differences between the observation laws. A direction in the operator’s kernel can have positive Fisher cost and zero first-order identifying signal. Orthogonal projection gives the cost spent in that direction. This answers a different question from how far apart the observed laws ended up. Table 1 shows why that distinction matters for exploration diagnostics. All six alternations spend the same budget, 6.0714, and accumulate the same average pairwise separation energy, 1.5949. For two policies, the hardest-pair energy is zero and the reported errors are 50.7% and 50.3%. For the other four, the hardest-pair energy is 1.196 and reported error ranges from 11.3% to 13.9%. That table tests the failure of aggregate diagnostics. The Fisher result explains how costly motion can leave a distinction unresolved.

2. Can someone actually measure it? Appendix L tests a plug-in estimator that sees actions and observations from a known hypothesis class with an identifiable exposure-decay model. Across six policies, four environments, and 2,000 runs per policy–environment combination, it recovered the oracle policy ordering in this test (Kendall τ=1.0\tau=1.0). The median relative error in the estimated capacity varied from 18% to 35% across policies. This is evidence for ranking policies under that model, with substantial error in the magnitudes. The LLM intervention uses a different kind of access. We hold the responder fixed and query it under alternative policies to measure directional separation directly. Those queries yield unnormalized separation energy.

3. What do the experiments validate, and what remains open? The matched experiment deliberately fixes the budget and average separation. The comparison therefore tests whether those summaries can miss an unresolved pair. In the LLM intervention, the blind policy spent 44% more motion and obtained 0.4% as much separation energy as the targeted policy. Its observed identification error was 34.3%, compared with 0% for the targeted policy. The responder already exhibits differences between types at a fixed policy, so the intervention tests the directional mechanism outside the no-static-signature assumption. The human preference logs give a narrower result. The length-only and full preference models were the least separated pair in 18 of 19 batches. We take these results as evidence that measuring directional and pairwise separation can reveal what motion summaries miss. Direct empirical validation of the normalized capacity–waste split remains open.

Thank you for raising these two concerns. We have clarified the novelty claim and added an experiment that tests the decomposition directly.

Novelty. The contribution is a characterisation of the decomposition. In the local Fisher geometry, the weight on policy motion is the squared cosine of its angle to the identifiable subspace. The exponent of two follows from measuring the identifiable component by its squared norm. Under the respective axioms for cost and separation, Čencov’s theorem characterises each term up to scale. We have revised the manuscript to state this contribution more precisely.

Empirical validation. We now perturb a synthetic channel separately along the identifiable and null directions and compute the divergence exactly. Separation scales linearly with perturbation size along the identifiable direction, with a fitted log–log slope of 0.999. Along the null direction it scales quadratically, with a slope of 1.999. On the same channel, the orthogonal decomposition holds for the environment-averaged cost and fails for the supremum cost used to define the budget. This illustrates why the geometric result uses the averaged cost.

If these revisions address your concerns, please reconsider your rating.

Comments on the paper

1 comment

Here is the result we want to emphasize. Two policies can move just as much and separate the environments just as well on average, yet differ in whether they can tell a particular pair apart. In our matched experiment, each blind policy leaves two environments that generate the same transcript law. Even a learner that knows both laws must guess between them. The Fisher analysis shows how this can happen: some directions of policy change leave the observation laws unchanged to first order.

4PJg asked whether that effect survives outside a response model we designed. In the LLM intervention, we changed the policy and queried the same responder again. The blind policy spent 44% more motion than the targeted policy and produced just 0.4% as much separation energy. The revised proof of Theorem~1 uses the posterior at each step, and the multiclass result now has a separate argument for the divergence between transcript laws addressing iERn concerns.

DEMD asked the question that matters for using the diagnostic: would it change which policy you pick? In our four-policy comparison, the hardest-pair TV and Hellinger summaries were almost tied. At the largest tested budget threshold, ranking by separation per unit motion picked the policy with the lowest observed error, 4.2%. The highest error was 6.5%. The information-gain policy did best at the smallest threshold, and neither preregistered experiment supported using the same idea to split a budget across systems. The practical case we make is specific. When candidate policies achieve similar separation at different costs, measuring both can help a practitioner choose among them.

Reviewer concernHow we address itWhere
4PJg, DEMD · What does the decomposition add beyond separation?A policy change can have positive Fisher cost and zero first-order identification signal. In Table 1, the six policies have identical budgets and average pairwise separation, yet the two with zero hardest-pair separation have reported errors near 50% and the other four have errors of 11.3–13.9%.§3.2, Table 1 and Appendix Q
4PJg, DEMD Q2–Q3With a known hypothesis class and a specified exposure-decay model, the plug-in estimator recovered the oracle policy ranking in the synthetic test, with median relative errors of 18–35% across policies. In the LLM test we query the response under alternative policies and measure unnormalized separation directly.§2, §4.3 and Appendix L
4PJg, DEMD · Do the experiments validate the split?The LLM blind policy spent 44% more motion and produced 0.4% as much separation energy as the targeted policy. In the human preference logs, the same model pair was least separated in 18 of 19 batches. These results test directional separation and pairwise discrimination.§4.2–4.3 and Appendix X.6
iERnThe proof bounds information directly by accumulated endpoint separation, using the posterior predictive mixture at each step. The drift comparison in Assumption 5 enters only when we convert separation to capacity. We assume the channel constant for that conversion and state the additional regularity needed for a quadratic reverse-Pinsker bound.Theorem 1, Corollary 5 and Appendices C–E
iERn, DEMD Q6We bound directed transcript KL for a fixed pair before optimizing over pairs. Average pairwise capacity controls an aggregate information bound, while one indistinguishable pair alone forces 50% conditional Bayes error under equal prior odds.§4.2, Remark 18, Theorem 33 and Corollary 34
iERn · Can readers tell what each result assumes and what each experiment tests?Readers can now look up the assumptions for each result and trace how the matched policies were constructed and evaluated. For the local geometry, we explain that the projection follows from the derived metric and that a quantitative lower comparison requires a lower singular-value bound.§3.2 and Appendices B–C, K.18
DEMD Q1At the largest budget threshold in the four-policy comparison, efficiency ranked the policy with the lowest observed error first. The accumulated TV and Hellinger summaries were nearly tied. Expected information gain did best at the smallest threshold, and both cross-system allocation experiments missed their preregistered success criteria.Appendix W and W.1
DEMD Q4–Q5Across all 32 policies, the in-sample R2R^2 is 0.874 for separation energy and 0.259 for motion budget. The Gaussian calculation checks reverse-Pinsker regularity in that model. The observed separation minimum describes the sampled runs. The almost-sure floor, drift regularity, and operator conditioning require separate justification.§4.1 and Appendix P
DEMD Q7Both methods can be posed using a predictive law that accounts for how the responder reacts to a proposed policy. Our analysis explains which policy changes produce little separation under that law.§5 and reply below

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.