acceptodds
Under review as a conference paper at ICLR 2027

Policies Are Programs: A Factorial Theory of Resource-Coupled Control

Abstract

A deployed controller changes its environment through both its actions and the resources used to compute them. Reconstructing its value from separate action and execution evaluations assumes that these channels combine additively. We characterize the resulting error by an execution–actuation factorial contrast. An exact Bellman identity separates local interaction from cross-channel propagation: even an additive one-step kernel can have nonadditive long-horizon value. Discount- and mixing-dependent bounds have asymptotically sharp leading constants, attained by two-state constructions. A four-state lower bound further shows that arbitrarily many complete trajectories from the three separate evaluation arms can leave the sign of deployment benefit unidentified, even with additive kernels. An exact stationary queue law identifies when the additive model rejects a beneficial deployment, including families with vanishing execution loss and unbounded rejection regret. Reanalysis of held-out synthetic queue records finds a predicted gain of versus an actual gain of at the same operating point. A randomized-block CPU control independently exhibits interaction with action-release latency removed. Together, these results characterize the accuracy, information requirements, and decision consequences of additive deployment evaluation.

Primary area:
reinforcement learning
Keywords:
resource-coupled control, policy evaluation, causal interaction, Markov decision processes, queueing theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_44XsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies deployment evaluation when a controller affects its environment through both its actions and the computational resources consumed in producing those actions. It formalizes the error of reconstructing coupled deployment value from three separate evaluation arms as an execution–actuation factorial interaction. The paper derives an exact Bellman decomposition, discount- and mixing-dependent bounds with asymptotically sharp constants, and a lower bound showing that complete trajectories from the three separate arms may not identify even the sign of deployment benefit. An analytically tractable queueing example characterizes false deployment rejection, while synthetic queue experiments and a CPU-sharing microbenchmark demonstrate substantial interaction and several decision reversals.

Critical strengths

1.Strong theoretical treatment of interaction and non-identifiability. The Bellman decomposition clearly explains why locally additive transition perturbations need not yield additive long-horizon value. The sharp propagation bounds and the three-arm trajectory indistinguishability result are technically substantive.

2.Careful experimental methodology and reporting. The four interventions are defined explicitly, paired randomness is used appropriately, and the paper carefully distinguishes closed-loop from frozen-tape estimands. The authors are also transparent about post-hoc comparisons, null results, hardware uncertainty, and the limits of the queueing divergence result.

Critical weaknesses
  1. The practical target is insufficiently established. The paper is motivated as addressing an important deployment-evaluation practice, but it does not demonstrate that researchers commonly estimate joint deployment value using J11=J01+J10−J00J_{11}=J_{01}+J_{10}-J_{00} or an equivalent transport assumption. Direct end-to-end evaluation of J11J_{11}, separate reporting of performance and computation, and Pareto evaluation are not subject to the same reconstruction error. Without stronger evidence about the prevalence of the targeted workflow, the practical significance of the theory remains unclear.

  2. The experiments have limited external validity for learned policies. The empirical evidence consists of synthetic threshold/periodic controllers, a queue simulator designed around action–capacity coupling, and one short-window Python CPU microbenchmark whose active controller consumes roughly 30% of the measurement window. There is no trained policy, neural inference workload, production system, or small-footprint hardware experiment. The experiments convincingly demonstrate possibility, particularly near capacity boundaries, but do not show that the effect materially alters representative RL evaluations.

Questions for authors
  1. What concrete existing evaluation workflows use the three-cell additive reconstruction, or make an equivalent assumption that the execution effect measured under suppressed actuation transfers unchanged to the actuated regime? Please distinguish this from papers that merely report task performance and runtime separately, or that directly measure the complete deployed system. Evidence that the targeted practice is common would materially strengthen the paper’s motivation and impact.

  2. Do the authors have evidence that the reported interaction remains decision-relevant for a trained policy and a realistic inference footprint?

  3. What is the intended practical use of the proposed bounds when kk, ϵC\epsilon_C, ϵU\epsilon_U, and the mixing coefficient α\alpha cannot be reliably estimated from ordinary deployment logs? Is there a realistic setting in which these quantities can be certified more easily than directly evaluating J11J_{11}, or should the bounds mainly be understood as structural rather than operational results?

Reviewer_787YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies controllers that share computing resources with the system they control, and asks whether their value can be predicted by measuring the benefit of their actions and the cost of their computation separately, then adding the two. The authors show that this shortcut can fail, supported by a clean decomposition of the error, sharp bounds, an exact queueing analysis, and experiments in simulation. The work is rigorous and transparent, and the math appears sound as far as I can tell. My main concern is significance. The key takeaway, that a controller should be evaluated as it is actually deployed, feels close to common sense, and the evidence comes from simple hand-designed controllers in heavily loaded queues rather than learned policies. I may be missing part of the motivation, and a convincing example with a learned controller would help me see the broader relevance for ICLR.

Critical strengths
  1. The core decomposition is clean and insightful. I found the framing of the additive shortcut's error as a factorial interaction simple and convincing. The exact recursion makes it easy to see how the two effects build on each other over time. The small two-state example, where an additive transition law still gives nonadditive value, is where the idea really clicked for me. As far as I could check, the main results appear correct.
  2. The experimental design is careful. I appreciated how much care went into the experiments. Shared randomness across arms, replay controls, and null checks give me confidence that the reported effects are measured fairly. I also liked that every configuration is reported, including the ones with no effect.
  3. The examples are clear and well chosen. The worked examples, especially the exact queueing case, made the mechanism concrete for me and helped me follow the more abstract results. I also liked that the paper anticipates natural objections about what each of the four cells is measuring.
Critical weaknesses
  1. My main concern is that the core message may be close to common sense. If running a controller affects the system it controls, it seems natural to expect that its decision quality and its compute cost cannot simply be measured separately and added. The paper formalizes this carefully, but I am not sure the conclusion changes what practitioners would do, which is to evaluate the deployed configuration. The tools also look fairly standard to me: the four-cell contrast, a recursion obtained by subtracting Bellman equations, and span and Dobrushin bounds. It would help if the authors could point to settings where the three-cell reconstruction is actually used to evaluate learned controllers, and show that the interaction changes a decision there. I may be underestimating how common this shortcut is, so clarification on this point would be valuable.
  2. I am not sure the three-arm information limit goes beyond a coverage issue. If I understand the construction in Theorem 5 correctly, actuation acts only at a state that is reachable only through execution, so the separate arms never visit the region where the two channels meet. The paper notes that targeted transition queries, or knowledge of the three kernels, would identify the coupled kernel in this case. This makes me think the result is mainly about a sampling design without coverage, which seems narrower than the abstract suggests. A positive counterpart, for example showing when the separate arms do suffice under a coverage condition, would make this part more useful.
  3. It is unclear to me how the bounds would be used in practice. Theorem 2 depends on kernel discrepancies and mixing coefficients that, as the paper acknowledges, ordinary logs may not identify, and the decision-margin certificates need model bounds BiB_i. I could not find a discussion of where these would come from. An example of estimating or bounding them from data, or a case where the bound changes a decision without running the coupled cell, would help.
  4. The experiments are fairly narrow in scope. The controllers are fixed threshold gates rather than learned policies, and the capacity loss is a synthetic coordinate. The physical experiment uses short 0.3-second windows in a shared container. I also noticed that in Table 4 the reference cell at nominal load 0.95 differs between the busy-work and idle rows, even though the controller runs off-core in that cell. An explanation and per-cell intervals would help.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.