- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that standard exploration diagnostics (action entropy, switching, drift) measure policy motion, not whether that motion stays informative once data-generating agents respond.
It factorises exploration into a KL-drift cost Δ_t and a strategic observability O_t, defines strategic capacity C = Σ O_t^² Δ_t, and splits the information budget exactly as SIB = C + W, with efficiency η = C/SIB.
The Theory: a binary-hypothesis information bound: a.) Theorem 1 with a truncated capacity floor, local Fisher geometry splitting motion into identifiable and null components b.) Theorem 2. , Chentsov-type characterisations, a frontier sandwich, and Fano corollaries; a commons model yields a parameter-free log-precision prediction (½ log N) . c.) Experiments: a budget-matched mirage where six policies identical in every motion diagnostic separate only through minimum pairwise separation (Tab. 1), an interventional test with a real LLM responder where blind outspends targeted by 44% of motion for 0.4% of the separation (Tab. 2), a recommender collapse experiment, and a weak-signal replication using human preference data.
Critical strengths
The paper identifies a genuine gap and demonstrates it with a sharp instrument!
Equal-budget policies spanning the whole error range while identical in entropy, switching, exposure, and average separation is exactly the kind of matched counterexample that makes a diagnostic claim hard to dismiss, and the interventional LLM test goes beyond author-designed response maps in at least one system.
Very direxct and explanibale elements and real limitations comments.
The paper flags that ĈT is an unnormalised surrogate because C_up is unidentified, admits Assumption 2 fails globally in its own headline counterexample, reports the degraded fit in the weak-signal replication with intervals, and includes an AI-use statement.
Critical weaknesses
2 critical comments
The central decomposition is an accounting identity: O_t is defined to make SIB = C + W hold, so the only measured quantity is accumulated squared separation; C_up is never identified, leaving "capacity" and "efficiency" η as unnormalised surrogates. No decision problem shows that factorising SIB changes outcomes versus accumulated separation, EIG, IDS, or a Hellinger/Chernoff tracker; since the mirage turns on the minimum pairwise separation, which the framework does not foreground, theory and ad-hoc summary are confounded.
Many empirics largely confirm built-in structure: author-designed response maps, a known parametric estimator, a by-construction counterexample, and a lone external test that fits worse without analysis. The headline R² = 0.92 vs 0.07 leaves unit counts, held-out status, and the ĈT > 0.01 threshold rule undiscussed, with no error treatment for the regression (unlike in Table 1). Assumptions 2, 5, 6 and the ess-inf floor are never checked empirically.
Questions for authors
-
In any decision problem wheter it is policy selection, stopping, or budget allocation across systems, does the capacity C_T outperform accumulated squared separation ĈT, an EIG/BALD-style criterion, IDS, or a worst-pair Hellinger tracker? If not, can you explain waht the factorisation add?
-
Can C_up be estimated or bounded in even one system, and how do the reported rankings and R² change under plausible misspecification of it?
-
What formally distinguishes a "strategic" response from a generic policy-dependent kernel ?
-
For the R² = 0.92 vs 0.07 results: how many units, is evaluation held out, and how sensitive is the fit to the ĈT > 0.01 threshold used to select these 29 policies? Could report and comment the same regression without thresholding.
-
Could all or some of the sssumptions 2, 5 and 6 be checked empirically, and how non-vacuous are the constants? In particular, report empirical lower quantiles of S_T and the conditioning of J_t in the RLHF and recommender systems, where geometric freshness decay should stress the reverse-Pinsker regularity.
-
The mirage is decided by minimum pairwise separation while the theory centres on summed separation: is there a version of the theory that would predict the binding pair, or is this an unexplained regularity of the construction?
-
in a completely different fields and within very different contexs sinfotaxis (for olfactory searches) and infotaxis derived strategies for multi armed bandit have demonstrated efficient policy and strategy decision. Can you relate your work to these ?
2 comments
The revision addresses all your concerns and includes two new results.
Q1. Policy selection and budget allocation. Within a system, accounting for cost changes the policy ranking. The four informative policies share the same realised separation sequence on their hardest pair. Their accumulated total-variation and Hellinger scores agree to within a relative spread of , leaving them effectively tied on both measures. At the largest budget, the efficiency criterion selects the policy with the lowest observed error (4.2%, compared with 6.5% for the highest-error policy). The Spearman correlation between efficiency and error is .
Q2. Local constants. In the local regime, the averaged cost is . For two nonnegative costs, their maximum lies between their mean and twice their mean. It follows that is the largest eigenvalue of , while is half its smallest nonzero eigenvalue. Their ratio is therefore twice the condition number on the nonzero eigenspace. The numerical tightness ratios are 0.989 and 1.000, respectively. Applying Weyl’s inequality to a 120-point -net certifies the uniform bound . A denser 7,132-point grid gives a reference value of 8.1006. We also show that the direct plug-in estimate approaches the constant from below and therefore cannot certify an upper bound.
Q3. Bandit comparison. Section 2 now explains the distinction using the identity-response bandit setting. With an identity response map, policy changes are transmitted directly, so the strategic response cannot erase separation.
Q4. Full-sample results. The main text now reports the fits across all 32 policies first. The values are 0.874 for separation energy and 0.259 for the budget. Both describe in-sample variation across policies.
Q5. Empirical constant and decay. The empirical constant is 4.2912, at the upper endpoint of the analytically derived range. As decay increases, the ratio moves toward , making the assumption easier to satisfy in this setting.
Q6. Zero capacity. The revised result gives an exact equality. At zero capacity, the two transcript laws are identical, so the transcript carries zero mutual information about the environment.
Q7. Infotaxis. We explain how infotaxis selects actions by expected information gain in the identity-response setting. We also report that it outperforms our efficiency criterion at the smallest budget tested.
If the revisions resolve your concerns, please reconsider your rating.
Thank you for comments. Please see the general rebuttal and specific responses below.
1. Does the factorization improve a decision? At the largest threshold we tested, yes. Appendix W compares four policies in the same synthetic system. Their accumulated hardest-pair squared TV and Hellinger summaries are nearly identical. The spreads are 1.7×10−5 and 1.3×10−5, respectively. Their full-horizon separation per unit motion differs by a factor of 1.5. At the first step reaching the threshold B=8, the efficiency ranking matched the observed error ranking. The policy ranked first by efficiency, ε-greedy, had 4.2% observed error, compared with 6.5% for the highest-error policy at that threshold. At B=2, the posterior-guided policy, which uses expected information gain, had the lowest observed error. Appendix W reports all four thresholds and defines the evaluation at the first crossing, which can include overshoot. The Hellinger comparison uses accumulated summaries rather than a separate adaptive Hellinger policy.
Across systems, both preregistered allocation experiments missed their success criteria. In the first, allocation by separation per unit cost beat allocation by accumulated separation at three budgets and lost at the fourth. The largest difference occurred near a discrete jump in the error curve. In the second design, which reduced those jumps, equal splitting had the lowest observed mean error at all four budgets. At B=40, mean errors were 0.0760 for equal splitting, 0.0782 for accumulated separation, 0.0845 for separation per unit cost, and 0.0770 for the model-weighted oracle.
2. Can Cup be estimated or bounded? We have no empirical estimate of Cup for the reported systems. The fitted predictor is unnormalized separation energy. Multiplication by a common positive constant within one system would preserve its rank order and the R2 of the same intercept-including regression on the same units. An absolute cutoff would need to be rescaled as well.
3. What makes the response strategic? Our model specifies a response map from the learner’s announced policy or continuation and the latent environment to the state entering the observation kernel. Marginalizing that state gives a policy-dependent kernel. A generic policy-dependent kernel can represent the same observable law. The response map matters when we ask what the responder would do if the learner announced a different policy. Standard bandit observation laws can also differ greatly in how informative their actions are.
4. How many units and how sensitive is the fit? We fit the regression to one outcome per policy in the synthetic RLHF sweep. With all 32 policies, R2 is 0.874 for separation energy and 0.259 for the motion budget. The cutoff of 0.01 retains 29 policies and yields 0.917 and 0.068, respectively. Every cutoff from 10−6 to 0.05 retains the same 29. All three omitted policies have zero separation energy and error 0.5. The revised main text leads with the all-policy fit.
5. Do the assumptions hold and are the constants informative? In the synthetic Gaussian model, the shared-variance formula gives KL/TV2 between π and 4.2912 on the specified range. As the mean gap decays, the ratio approaches π. The global no-static-signature condition fails in the matched example and in the LLM intervention. For each of the two blind policies in the matched example, the relevant response laws agree on its unresolved pair. Across 2,400 synthetic runs, the smallest observed mean-pair ST was 0.7974. Minimum-pair separation was zero in one third of runs and had median 2.3923 in the remainder. The observed minimum describes these runs. The theorem’s essential-infimum condition concerns all histories up to a null set.
6. Why does the minimum pair matter? Each blind policy leaves a pair that produces exactly the same transcript law. Conditional on that pair and with equal prior odds, the best possible error is 50%. The simulated errors were 50.7% and 50.3%. The other pairs can contribute plenty of separation to the average without helping the learner tell these two apart.
7. How does this relate to infotaxis and IDS? Infotaxis chooses moves by expected information gain about a source. IDS balances information about the optimal action against expected regret. In our setting, either approach needs a predictive law that accounts for how the responder reacts to each candidate policy. The local analysis characterizes directions in which that law supplies little or no discrimination per unit motion. Infotaxis · Information-directed sampling