Where Should Scarce Action Labels Go? Offline RL from Action-Free Building Logs

Offline reinforcement learning assumes logged state–action–reward transitions, yet across the physical world the action channel can be missing entirely. Legacy building automation is a canonical case: years of sensor data exist, but the control commands that produced them need not have been kept. One property makes this regime tractable: in physical control the reward is a function of the observation, so the data are action-free yet reward-complete and policy improvement rather than imitation is possible. Deploying on real actuators still requires some labeled commands, which raises the practical question of where a small acquisition budget should be spent in the pipeline. We answer it with a matched benchmark over twelve building×climate EnergyPlus configurations in which three placement strategies—imputing the missing column with an inverse model, grounding a latent policy learned from the action-free corpus, or supervising the latent during representation learning—are trained from identical labeled transitions and matched seeds. Where the labels enter matters most when they are scarce: at a 5% budget the three recover 36%, 59% and 89% of an oracle’s improvement, while offline RL on the labels alone falls below the logging policy. On this benchmark’s logs the two late placements converge by a fifth of a year; the early one keeps a lead over grounding-only at every budget, though the full-year margin does not survive resampling by building archetype. Decomposed into physical quantities, the recovered improvement is almost entirely a reduction in thermal discomfort, with no consistent energy savings. The early placement is also the one that holds up when the conditions change: it stays ahead under a biased, rarely adjusted logging controller, under labels collected in seasonal campaigns, and across full-pipeline redraws, whereas whether the two late placements converge depends on the logging condition.