Many real-world control logs, such as building sensor records, capture what happened to temperatures and energy use but not the control commands that caused it, so offline reinforcement learning cannot use them directly. This paper studies that setting for heating and cooling control, where the reward can still be computed from the logged readings alone, even without knowing what commands were issued. The paper's pipeline has three stages. Stage 1 learns a "latent action," a stand-in for the real command, purely from how the sensor readings changed, using no real commands at all. Stage 2 runs offline RL treating that latent action as the action, using the reward computed from the sensor readings. Stage 3, called "grounding," trains a decoder to map the latent action into a real command, since an actuator cannot act on a latent action directly, and this step needs real labeled examples. Deploying a policy therefore needs some real labeled commands, so the question is where a small labeling budget should go. The options are to supervise the latent action during Stage 1 so it lines up with real commands from the start, to hold the labels back for Stage 3 and use them only to ground the latent action at the end, or to skip the latent action altogether and impute, meaning fill in, the missing real command at every point in the logs by fitting a separate model, before running ordinary offline RL. Testing twelve simulated building and climate setups with matched data and budgets, the authors find that this choice matters most when labels are scarce. In that case, supervising the latent action during Stage 1 comes much closer to the improvement a policy trained with complete real commands would achieve than either grounding only or filling in the missing commands directly. Most of this gain comes from less occupant discomfort rather than energy savings, and the Stage 1 placement also holds up best when the data-collection conditions change, such as when the original controller behaves unusually or when labels are collected in short seasonal batches rather than spread through the year.
Response to Reviewer n7gs
We thank the reviewer for the review. Regarding the link to limited data: in our setting, the abundant data are the action-free logs, which contain one year of sensor readings for each building. The limited data are the action labels. A policy for real actuators needs some labels to map its decisions to commands. In an existing building, such labels are collected only after command logging is enabled, so only a few labels are available in the first weeks. The paper studies how to use these few labels together with the large action-free logs, and how long and in which seasons they should be collected.
Regarding the structure: in the revision, the related work (§2), problem formulation (§3), method (§4), experimental setup (§5) and results (§6) are presented in separate sections, and a new Figure 1 (page 2) shows the compared strategies before any notation is introduced. If there are specific points that the reviewer found unclear or unconvincing, we would be glad to answer them during the discussion period.