acceptodds
Under review as a conference paper at ICLR 2027

Where Should Scarce Action Labels Go? Offline RL from Action-Free Building Logs

Abstract

Offline reinforcement learning assumes logged state–action–reward transitions, yet across the physical world the action channel can be missing entirely. Legacy building automation is a canonical case: years of sensor data exist, but the control commands that produced them need not have been kept. One property makes this regime tractable: in physical control the reward is a function of the observation, so the data are action-free yet reward-complete and policy improvement rather than imitation is possible. Deploying on real actuators still requires some labeled commands, which raises the practical question of where a small acquisition budget should be spent in the pipeline. We answer it with a matched benchmark over twelve building×climate EnergyPlus configurations in which three placement strategies—imputing the missing column with an inverse model, grounding a latent policy learned from the action-free corpus, or supervising the latent during representation learning—are trained from identical labeled transitions and matched seeds. Where the labels enter matters most when they are scarce: at a 5% budget the three recover 36%, 59% and 89% of an oracle’s improvement, while offline RL on the labels alone falls below the logging policy. On this benchmark’s logs the two late placements converge by a fifth of a year; the early one keeps a lead over grounding-only at every budget, though the full-year margin does not survive resampling by building archetype. Decomposed into physical quantities, the recovered improvement is almost entirely a reduction in thermal discomfort, with no consistent energy savings. The early placement is also the one that holds up when the conditions change: it stays ahead under a biased, rarely adjusted logging controller, under labels collected in seasonal campaigns, and across full-pipeline redraws, whereas whether the two late placements converge depends on the logging condition.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, action-free reinforcement learning, latent action learning, learning from observation, semi-supervised reinforcement learning, action imputation, label-efficient learning, building control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_n7gsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Dear authors. From what I could understand, the work aims to tackle a very interesting problem of inferring actions from logged data, where actions are missing but the transition still contains (s,s′,r)(s, s', r). While I believe that Offline RL would benefit from such an approach to learn from inferred action data, this is not necessarily related to the problem of learning from limited data. Overall, the paper needs substantial work in splitting the problem formulation, the method, related work, and the scenario chosen for validation.

Critical strengths

None

Critical weaknesses

I'm confident the work isn't ready for publication.

Questions for authors

I don't have any questions.

1 comment
AuthorsOpenReview ↗

Response to Reviewer n7gs

We thank the reviewer for the review. Regarding the link to limited data: in our setting, the abundant data are the action-free logs, which contain one year of sensor readings for each building. The limited data are the action labels. A policy for real actuators needs some labels to map its decisions to commands. In an existing building, such labels are collected only after command logging is enabled, so only a few labels are available in the first weeks. The paper studies how to use these few labels together with the large action-free logs, and how long and in which seasons they should be collected.

Regarding the structure: in the revision, the related work (§2), problem formulation (§3), method (§4), experimental setup (§5) and results (§6) are presented in separate sections, and a new Figure 1 (page 2) shows the compared strategies before any notation is introduced. If there are specific points that the reviewer found unclear or unconvincing, we would be glad to answer them during the discussion period.

Reviewer_CA7gOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Many real-world control logs, such as building sensor records, capture what happened to temperatures and energy use but not the control commands that caused it, so offline reinforcement learning cannot use them directly. This paper studies that setting for heating and cooling control, where the reward can still be computed from the logged readings alone, even without knowing what commands were issued. The paper's pipeline has three stages. Stage 1 learns a "latent action," a stand-in for the real command, purely from how the sensor readings changed, using no real commands at all. Stage 2 runs offline RL treating that latent action as the action, using the reward computed from the sensor readings. Stage 3, called "grounding," trains a decoder to map the latent action into a real command, since an actuator cannot act on a latent action directly, and this step needs real labeled examples. Deploying a policy therefore needs some real labeled commands, so the question is where a small labeling budget should go. The options are to supervise the latent action during Stage 1 so it lines up with real commands from the start, to hold the labels back for Stage 3 and use them only to ground the latent action at the end, or to skip the latent action altogether and impute, meaning fill in, the missing real command at every point in the logs by fitting a separate model, before running ordinary offline RL. Testing twelve simulated building and climate setups with matched data and budgets, the authors find that this choice matters most when labels are scarce. In that case, supervising the latent action during Stage 1 comes much closer to the improvement a policy trained with complete real commands would achieve than either grounding only or filling in the missing commands directly. Most of this gain comes from less occupant discomfort rather than energy savings, and the Stage 1 placement also holds up best when the data-collection conditions change, such as when the original controller behaves unusually or when labels are collected in short seasonal batches rather than spread through the year.

Critical strengths

None

Critical weaknesses

Weaknesses

  • The paper reads as AI-written and is hard to follow throughout, for example "Huber loss enters only through the physics residuals of Appendix G" and "exogenous innovation."
  • The motivating claim, that real buildings keep sensor logs but not setpoint commands, has no supporting citation, and in practice setpoints are commonly recorded.
  • Since HVAC actuators are usually driven by PID controllers that reach a commanded setpoint within one or two steps, recovering the setpoint from sensor dynamics is easier than the paper suggests.
  • It is unclear why action labels would be scarce at all, since getting real actions does not require installing new hardware per actuator and logging could simply be enabled going forward rather than limited to a small budget, and in rare cases where actuators cannot reach a requested setpoint, it is unclear that even a human could label the true action after the fact.
  • Novelty is unclear. The differences claimed against closely related work, such as the labeled fraction swept or the offline RL algorithm used, read as incidental choices rather than a clearly stated new contribution.
Questions for authors
  • Can you cite evidence, such as a survey of building automation data practices, that commands are often missing from retained logs while sensor data is kept?
  • The imputation baseline alone recovers most of the oracle's improvement once given at least a fifth of a year of labels. Why frame recovering the action as a hard identifiability problem rather than an ordinary small-sample learning problem?
  • Since the labeled pool is generated as a full year and then subsampled, what in the pipeline actually represents the cost of a short real-world instrumentation campaign, rather than a researcher withholding already-collected labels?
  • In cases where an actuator cannot reach its commanded setpoint, how would a real deployment obtain a ground-truth label for that transition?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CA7g (1/4)

We thank the reviewer for the detailed questions. We answer them in the following order: the problem setting (Q1/W2, W4), how the labels are collected (Q3), what the sensor readings show about the commands (Q4, W3, Q2), and the contribution and the writing (W5, W1).

Q1 / W2: Are commands really missing from retained logs? We could not find a survey that measures how often commands are missing, and the revision does not claim that they are usually missing. Our claim is that such logs exist, and that every building is in this situation for some time after command logging is enabled (§1, lines 92–97; see W4). We give three kinds of evidence.

  • First, the building that motivated this work is an example. Its building automation system has kept sensor trends for years. Setpoints are entered at the DDC controllers, which run their own control logic, and the history of the entered setpoints is not kept (§1, lines 40–43).
  • Second, several causes have been reported in practice, and the revised introduction gives five of them with sources (§1, lines 45–79). A data historian is an optional part of a building automation system, and in a survey of 676 non-residential buildings, about a quarter of the buildings stored no monitoring data at all. Control parameters inside local controllers are not always exposed to the building automation system. Trend logging has to be set up for each point and has long been limited to a few high-level points. Trend data are overwritten unless they are archived. Security guidelines restrict connections between the control network and other networks.
  • Third, widely used public building datasets contain meter readings and weather data without control commands, for example BDG2 (3,053 meters in 1,636 buildings).

We agree that many recent systems do record setpoints, and the revised introduction now states this (§1, line 45).

W4: Why would labels be scarce, if logging can simply be enabled? In our setting, the labels are the commands that are logged after command logging is enabled. The budget counts the commands logged so far: at the 15-minute control step, 5% is 1,751 commands, which corresponds to 18 days of operation, and 20% corresponds to about 73 days (§5, lines 340–346; Appendix A, "Labeling budget"). Commands logged from a given day form one continuous window, and the even sampling in the main table (Table 1) is the most favorable way to have the same number of labels (§5, lines 342–343). Thus, the practical question is how long the commands should be logged before a policy is trained, and the new results answer this question in terms of season coverage (§6.2 and Table 2; see Q3). With a single window of 73 days, starting in January or in July, every strategy is still below the logging policy on average. With four 18-day campaigns, one in each quarter, the recovery is within 3 to 12 points of even sampling. Therefore, the labels need to cover the seasons in which the policy will be used, either by logging for a longer time or by running short campaigns in each season. A commissioning campaign that varies the setpoints on purpose gives more informative labels, and for this reason our labeled data use a wider perturbation than the log (σ = 2.5 instead of 1.0). However, in an occupied building, such a campaign reduces comfort and cannot run for a long time. The limitations section now states this difference (§7, lines 499–500).

AuthorsOpenReview ↗

Response to Reviewer CA7g (2/4)

Q3: What represents a short campaign rather than withheld labels? We agree with this comment. The main table samples a fixed number of labels evenly over the year (at 5%, one label every five hours on average), which is the most favorable case for that number of labels. In the submission, the expression "eighteen labeled days' worth" made this setting look like a campaign, and we corrected it in the revision (§5, lines 340–346; Appendix A, "Labeling budget"). We have also run two schedules that keep the same number of labels but collect them as an operator would: four campaigns of equal length that start at the beginning of each quarter, and one window that starts on January 1 or at mid-year (July 2). All three strategies were run on all twelve configurations with three seeds. The table below gives the mean recovery [median], followed by the number of configurations (out of 12) in which the strategy outperforms the logging policy:

even samplingfour campaignsone window from January 1one window from July 2
5% (1,751 labels)4 × 4.6 days18 days18 days
supervised latent89.4 [93.0], 12−18.4 [18.9], 8−48.6 [−33.3], 5−133.6 [−105.0], 1
grounding-only58.7 [59.1], 12−62.6 [−47.3], 2−46.8 [−40.8], 3−164.3 [−151.8], 1
imputation36.5 [41.2], 11−253.8 [−185.1], 0−389.5 [−249.7], 0−332.0 [−312.4], 0
20% (73 days of labels)4 × 18 days73 days73 days
supervised latent86.8 [87.3], 1281.4 [86.0], 12−98.5 [−36.1], 5−80.6 [−92.2], 2
grounding-only71.6 [79.7], 1268.5 [76.3], 11−2.0 [25.8], 8−80.9 [−67.3], 5
imputation77.1 [83.0], 1165.2 [67.4], 12−181.4 [−127.3], 2−317.9 [−338.2], 2

The table shows three results. First, at 5%, no strategy is reliably better than the logging policy when the labels come from campaigns or from one window. These policies use 85–99% of the oracle's energy, but their comfort violation is 2.6 to 7.8 times that of the oracle (Appendix L). Second, at 20%, four campaigns of 18 days are within 3 to 12 points of even sampling, and supervised latent remains the best strategy (+12.8 [4.7, 25.1] over grounding-only and +16.1 [10.3, 22.8] over imputation). With a 73-day window, however, every strategy is still below the logging policy on average, whether the window starts in January or in July. Thus, the outcome depends on whether the labels cover the seasons, and not only on the number of labels. Third, the best strategy also depends on this coverage. When the labels cover the seasons, supervised latent gives the highest recovery. With one window, this advantage disappears. In the July window, the difference between the two latent strategies is not significant at either budget (+30.7 [−103.1, 147.7] at 5% and +0.3 [−56.1, 59.0] at 20%), and in the January window at 20%, supervised latent is worse than grounding-only (−96.5 [−247.9, −10.7]). When the labels are not evenly sampled, imputation is the worst strategy, except in the 20% campaigns, where it is comparable to grounding-only. In the schedules where imputation fails, the mean error of its pseudo-labels is 1.9–2.9 °C, compared with 0.8–1.0 °C in the other schedules (§6.3, lines 457–459; Table 25). In the revision, this table is included in the main text (Table 2), §6.2 discusses these results (lines 417–437), and Appendix L gives the values for each configuration. Even sampling is presented as the optimistic reference (§1, lines 119–120; §5, lines 342–343).

AuthorsOpenReview ↗

Response to Reviewer CA7g (3/4)

Q4: What is the label when an actuator cannot reach its setpoint? In our setting, the label is defined as the issued command, and it is not inferred from the states afterwards. A system that logs setpoints records them regardless of whether the zones reach them, and our labeled data record the issued commands in the same way. We agree that this is an important point, and we think that it supports logging the commands instead of inferring them. In our buildings, while the system is running, the zone that a setpoint acts on does not reach the setpoint and stays more than 0.5 °C away from it (an unmet setpoint) in about 12% of the steps on average, and in up to 55% of the steps in one configuration. A label inferred from the sensor readings cannot show how far the command was from the temperature that was actually reached. The revised problem formulation defines what a label is (§3, line 213), and §6.3 reports these shares (lines 447–449; Table 24 in Appendix M).

W3: Recovering setpoints from sensor dynamics. We measured this in all twelve configurations (§6.3, lines 440–464; Appendix M). The control loop responds quickly, but a setpoint change is not always visible in the readings within one control step. We consider steps in which a setpoint is binding, that is, the zone it acts on is within 0.5 °C of it while the system runs. When a binding heating setpoint is raised by 1 °C or more, the coldest zone is within 0.5 °C of the new value one step (15 minutes) later in 56% of the cases. When a binding cooling setpoint is lowered by the same amount, the warmest zone is within 0.5 °C of the new value in 62% of the cases. When the heating setpoint is lowered or the cooling setpoint is raised, the zones have to cool down or warm up on their own, and the shares are 21% and 36% (Appendix M).

A larger problem is that a setpoint often has no effect on the zones. When every zone is warmer than the heating setpoint by more than 0.5 °C, nearby values of the setpoint give the same readings, so its value cannot be recovered, however fast the control loop is. The same holds for a cooling setpoint that is more than 0.5 °C above the temperature of every zone. We call such a setpoint floating. On average, the heating setpoint is floating in 61% of the steps and the cooling setpoint in 53% (§6.3, lines 441–447). These shares rise to 82% for heating in the hot climate and 66% for cooling in the cool climate, and both setpoints are floating at the same time in 19% of all steps (7–42% depending on the configuration; Table 24 in Appendix M).

The inverse model used for imputation shows the effect of this. Its per-step error is 0.92 °C for heating and 0.87 °C for cooling at the 5% budget, and 0.80 and 0.72 °C with a full year of labels. A simple guess that knows only the monthly median of the logged setpoint, and nothing about the step, has an error of 0.72 and 0.71 °C. Even with a full year of labels, the inverse model is therefore not more accurate than this guess (§6.3, lines 453–456; Table 25). We think that the main difficulty comes from the floating setpoints rather than from the speed of the control loop.

Q2: Identifiability or small-sample learning? For the main result, which is the performance of the policy, small-sample learning is the better description, and the revision presents the claims in this way (§6.1, lines 373–375). However, the reason is not that the inverse model learns to recover the commands, because its per-step error is never lower than that of the guess described above. Two observations help explain why imputation still becomes competitive at larger budgets. First, a wrong value on a floating step is usually still consistent with the logged transition. At the 5% budget, the imputed value stays on the inactive side of every zone in 99% of the floating steps (§6.3, line 457). Second, at larger budgets, the labeled transitions themselves are also used in offline RL, which adds 12 points at 20% and 15 points at a full year (§6.3, lines 459–461; Table 14 in Appendix D). The revision makes no claim about identifiability (§2, lines 184–188) and reports these measurements as properties of the data (§6.3).

AuthorsOpenReview ↗

Response to Reviewer CA7g (4/4)

W5: Novelty. We do not claim a new method, a new setting or a new budget sweep. Our contribution is the set of findings from a matched comparison, which cannot be seen when each strategy is evaluated separately:

  1. The standard semi-supervised approach, which imputes the missing actions and then runs offline RL, performs poorly at small label budgets even when the labels are spread evenly (−27.0% at 1% and 36.5% at 5%; Table 1). Part of the reason can be measured. Without an output constraint, 65% of the pseudo-labels at 1% fall outside the range of the labels, and training the inverse model with the same recipe as the decoder adds 164 points at 5% (§6.1, lines 369–370; Appendix D). Imputation also performs worst in most cases when the labels come from campaigns or windows (§6.2; Q3), because a setpoint often has no effect on the readings (§6.3; W3).
  2. When the labels cover the seasons, supervising the latent actions with the labels gives the highest recovery: 89.4% at 5% with even sampling and 81.4% at 20% with four seasonal campaigns (Tables 1 and 2). In both cases, it outperforms the logging policy in every configuration. With labels from a single season, this advantage disappears (§6.2; Q3).
  3. With labels for a fifth of a year, either evenly sampled or from seasonal campaigns, the difference between imputation and grounding-only is not statistically significant (Table 1; §6.2, line 434).
  4. Almost all of the recovered improvement comes from lower discomfort, and there are no consistent energy savings (§6.4).

We agree that the differences from LAWM described in the related work look secondary. The revision states the contribution as the above findings (§1, lines 130–145) and shortens the comparison with LAWM (§2, lines 167–173).

W1: Readability. We agree with this comment. The sentence about the Huber loss quoted by the reviewer described a term of an appendix ablation in the main method section, and we removed it. The Huber loss now appears only in the ablation of Appendix G. "Exogenous innovation" is replaced by a plain description (§4, lines 258–260). The three strategies are defined once (§5, lines 331–338) and called by the same names throughout the paper, and the terms "placement", "early/late placement", "pipeline", "arm" and "rung" are no longer used. Tier A and tier B are explained where they are used (§6.5, lines 482–485). Most of the caveats that were placed within the results in the submission are now collected in the limitations section (§7, lines 496–521).

Reviewer_byykOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an extensive benchmarking in using three offline RL methods on two environments for energy systems with 12 types of buildings, different zones and 5 years of data. The main concern of this paper is that how to train the optimal policies where the action labels are scarce, and not always known. There is a limited budget in revealing them that should be used in an intelligent way. For this purpose, they used three already available methods:

  • predicting missing actions using inverse dynamic and then train offline RL. In their cased the offline LR method is implicit Q-learning (IQL)
  • learn latent actions without supervision, train implicit Q-learning and then decode the actions into physical actions
  • use action labels during latent representation learning, train implicit Q-learning and then decode into physical actions
Critical strengths
  • The main strength of the paper is its exhaustive set of experiments and results.
  • Another interesting point of this work is its application and its raised questions on energy systems and limited budgets on action labels.
Critical weaknesses
  • One of the concern with the paper is that, as a benchmarking paper, it is expected to be positioned in compare to other similar benchmarking. The paper can profit from a better discussion on the related benchmarking on energy systems with or without limited budget.
  • The paper has many experimental results, almost all presented in the appendix. For instance, the paper shows the best place to use the action label budget is during latent action representation learning, rather than just using this budget for action reconstruction or decoding. Better discussions and conclusions on each experimental section can make the results more valuable and usable for future works. For instance at the end of each subsection, the paper can discuss, what is the main achievement of this experiment and what is the main message.
  • This paper studies an interesting application of energy systems with limited budget of known action labels. Therefore, it has two important questions to tackle, which have been partially introduced in the paper. The energy systems are time series data, they have temporal and seasonal characteristics, which can be used/studied in better predictions of future actions, better representation learning of actions, or better adapting of offline RL methods on such type of data.
  • This paper requires more modifications on its writing and its structure design. The main part of paper can contain more experimental results which is the main contribution of the current benchmarking work.
Questions for authors

A suggestion rather a question:

Another interesting axis of this work is to formulate the optimization question of the available budget for labeling actions, with the objective of learning the optimal policy using offline RL algortihms. Is it possible to design a routing strategy on the available three methods to better use the current budget and better learning of the optimal policy.

3 comments
AuthorsOpenReview ↗

Response to Reviewer byyk

We thank the reviewer for the detailed reading. The summary in the review describes three offline RL methods on two environments, twelve building types and five years of data. This is different from our setup, and we think that the submission did not describe the setup clearly enough. Our setup uses one offline RL algorithm (IQL), which is shared by three ways of using the same labels, four building archetypes in three climates (twelve configurations), and one year of action-free data for each configuration. In the revision, this setup is described in the introduction (§1, lines 121–129) and in a new Figure 1 (page 2), and the experimental setup has its own section (§5, lines 313–353).

W1: Positioning against energy benchmarks. We added a related-work paragraph (§2, lines 190–197) and an appendix table (Table 26, Appendix N). The table compares building-control environments and datasets with three questions: whether they provide offline data, whether the data include the actions, and whether the number of action labels is varied. Sinergym, BOPTEST, CityLearn, Energym and BEAR are interactive simulators. In these simulators, the controller chooses every action, so missing actions do not occur. We use Sinergym to generate our logs and remove the actions from them. The Smart Buildings Control Suite (telemetry from 11 buildings over six years) and B2RL provide logged data from real buildings, and both include the issued setpoints. The most closely related offline evaluation on HVAC data, Wang et al. (2024), varies the size and quality of the dataset, but every transition has an action. To our knowledge, none of them varies the number of action labels, and this is what our study adds.

W2 and W4: Results in the appendix, takeaways, structure. We agree. The results section (§6) is now organized around five questions, and each question has its own subsection that ends with a one-sentence takeaway: which strategy makes the best use of evenly spread labels (§6.1), how the label schedule changes this result (§6.2), why imputation performs poorly (§6.3), what the improvement consists of (§6.4), and how stable the ordering is under full redraws, a biased logging controller, different reward weights and a different transition convention (§6.5). The additional tenth page is used for the new results on label schedules (§6.2, Table 2) and on the setpoints (§6.3).

W3: Temporal and seasonal structure. We agree, and the revision presents this as one of the findings (§1, lines 130–145; §6.2). The climate and the season determine which setpoint can be inferred from the data. For example, the heating setpoint is floating in 82% of the steps in the hot climate, and the cooling setpoint in 66% of the steps in the cool climate (§6.3, lines 441–447; Table 24 in Appendix M). In addition, the time when the labels are collected is as important as the number of labels. At 5%, four short campaigns reduce the recovery of every strategy by more than 100 points compared with even sampling. At 20%, four 18-day campaigns are within 3 to 12 points of even sampling, while one window of the same total length fails, whether it starts in January or in July (§6.2, lines 417–437; see also our reply to Reviewer CA7g, Q3). The method already uses temporal structure in two places. Stage 1 is conditioned on a one-hour history window (§4, lines 273–274), and the seasonal comfort bounds are given to the policy and the decoder as inputs (§4, lines 309–311). When these bounds were not given and the policy had to infer the seasonal change from calendar features, the policy smoothed the change, and the office building had most of its annual violation in the two shoulder months (Appendix A, "Exposing the comfort bounds"). Sequence models and history-conditioned critics are possible extensions, and we mention this in the limitations (§7, lines 514–515).

Question: Budget allocation as an optimization problem. In our setting, the labels are not consumed when a stage uses them. For example, supervised latent uses the same labels in Stage 1 and again in Stage 3. Therefore, we think that the question is which stages should use the labels, rather than how the budget should be divided among the stages. In our results, when the labels cover the seasons, using them both for representation learning and for the decoder is the best strategy at every budget up to a fifth of a year. At a full year, its difference from imputation is not statistically significant (§6.1, line 363). When the labels come from a single season, supervising the latent actions gives no benefit, and no strategy is reliably better than the existing controller. Therefore, the labels should be collected across the seasons before a policy is trained. The conclusion now states this as a decision rule (§8, lines 529–537). A hybrid strategy that combines imputation with supervised latent is left for future work.

I thank the authors for their detailed responses and revisions. I acknowledge that my concerns have been partially addressed, particularly regarding the experimental comparisons and seasonal characteristics of the data.

However, I have decided to maintain my original score. My main concern remains the clarity of the scientific narrative. Despite the revisions, the motivation behind the different experiments, their connections, and their contributions to the main conclusions are not sufficiently clear. Given that the paper's contribution is primarily empirical, I believe these aspects are essential and require further improvement before publication at ICLR.

AuthorsOpenReview ↗

Second revision: motivation, connections and contributions of the experiments

Thank you for your reply. We agree that the previous revision said what each experiment shows, but not clearly enough why each experiment is run, how it follows from the previous one, and what it contributes to the conclusions. We have uploaded a second revision that addresses these three points. No experiment was added or removed, and no result changed. Text changed in this revision is shown in blue, and the line numbers below refer to it (our earlier responses refer to the version of November 11).

  • The two questions. §1 now states the two questions of the paper (lines 95–97): where in learning the labels should be used, and how long and in which seasons they should be collected. The findings in §1 follow the order of §6, each with its subsection (lines 132–147).
  • How the experiments are connected. A new opening of §6 (lines 351–358) restates the two questions and says why each experiment is run and what it varies and keeps fixed. The titles of §6.1 and §6.2 now name the two questions, and each subsection starts from what the previous one leaves open (lines 368, 409–412, 418–420, 441–443, 467–470 and 485–494).
  • What each experiment contributes. The conclusion (lines 523–538) states which subsection supports each part of the decision rule, and what §6.4 and §6.5 add to it. The rule says when a policy can be trained and which strategy should use the labels, which is our answer to your suggestion about routing the budget among the strategies.
  • Appendix. A guide at the start of the appendix (page 15) gives, for each appendix, the question it answers and the part of the main text it supports.
ExperimentWhy it is runHow it follows from the previous oneWhat it contributes
§6.1 Where should evenly spread labels be used?To see only the effect of where the labels are used, they are spread evenly over the year, the most favorable schedule.Starting point: the strategy and the budget are varied.Supervised latent is best at every budget, and learning through latent actions beats imputation only when the labels are scarce (second part of the rule, part of the third).
§6.2 How long and in which seasons should the labels be collected?Labels accumulate from the day command logging is enabled, so even sampling is optimistic.Same budgets as §6.1; only the schedule changes.A single window fails even at 73 days, four 18-day campaigns come close to even sampling, and supervision helps only when the labels cover the seasons (first part, and the condition of the second).
§6.3 Why does imputation perform poorly?Imputation falls behind in §6.1 and §6.2 whenever the labels are scarce or do not cover the seasons, even in its strongest form.Explains this common result from the logged setpoints, without new training.A setpoint has no effect on the readings in more than half of the steps, and the inverse model is no better than a monthly guess (third part).
§6.4 What does the improvement consist of?Recovery is computed from a reward that adds an energy penalty and a discomfort penalty, so it does not show which of the two improved.Reports energy and discomfort for the rollouts of §6.1 and §6.2.The gains are in comfort, and the policies that fail in §6.2 use no more energy than the oracle but cause more discomfort (scope of the rule).
§6.5 How stable is the ordering?The answers in §6.1 and §6.2 rest on the ordering of the strategies, not on their absolute recoveries.Repeats the 5% comparison with redrawn data, other reward weights, another Stage 2 transition and a biased log.The ordering at 5% holds in all four checks, while the absolute values change by tens of points (how far the rule can be trusted).

We would be grateful if the revision could be taken into account in your final assessment, and we would be glad to answer any further question before the end of the discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

We thank all three reviewers for their comments. Three concerns were raised in more than one review: the paper was difficult to follow, the problem setting was not supported by evidence, and most of the results were reported only in the appendix. We have uploaded a revised version of the paper. The main changes are as follows.

  1. Structure. The related work (§2), problem formulation (§3), method (§4), experimental setup (§5) and results (§6) are now presented in separate sections. The results section is organized around five questions (§6.1–§6.5): which strategy makes the best use of evenly spread labels, how the label schedule changes this result, why imputation performs poorly, what the improvement consists of, and how stable the ordering of the strategies is. Each question has its own subsection, which ends with a one-sentence takeaway. A new Figure 1 (page 2) shows the two approaches and the three strategies without notation, and Figure 2 (§4) shows the details with notation.
  2. Problem setting. The introduction now gives evidence that building logs without command history exist, including the building that motivated this work (§1, lines 40–79). It also states that many recent systems do record setpoints (line 45). The label budget is now defined as the number of commands logged after command logging is enabled, and the label schedule as the period in which they are logged (§3, lines 213–215; §5, lines 340–346). The paper also states that spreading a fixed number of labels evenly over a year is the most favorable schedule (lines 119–120 and 342–343).
  3. New results in the main text. (a) In the submission, the label schedules were tested on one configuration. In the revision, they are tested for all three strategies on all twelve configurations at the 5% and 20% budgets (§6.2, lines 417–437, Table 2, Appendix L). With a single window of 18 or even 73 days, no strategy outperforms the logging policy on average, whether the window starts in January or in July. With four 18-day campaigns, one in each quarter, the recovery is within 3 to 12 points of even sampling, and supervised latent recovers 81% and outperforms the logging policy in all twelve configurations. Supervising the latent actions is helpful when the labels cover the seasons, but this advantage disappears when the labels come from one season. (b) For every configuration, we report how often each setpoint is binding, floating (it has no effect on the readings because every zone is on its inactive side) or unmet. We also compare the error of the inverse model with that of a simple guess that knows only the monthly median setpoint (§6.3, lines 440–464, Appendix M). Each setpoint is floating in more than half of the steps, and even with a full year of labels, the inverse model is not more accurate than this guess. Based on these results, the conclusion (§8, lines 529–537) now gives a decision rule for when and how to collect labels.
  4. Writing. The three strategies are called by the same names throughout the paper (imputation, grounding-only, supervised latent), and the terms "placement", "pipeline", "arm" and "rung" are no longer used. Most of the caveats that were placed within the results in the submission are now collected in the limitations section (§7, lines 496–521).

Because the paper was reorganized and the wording was revised for readability, the pdfdiff marks most of the text. The new results are in §6.2 (Table 2, Appendix L) and §6.3 (Appendix M), and the comparison with building-control benchmarks is in §2 (lines 190–197) and Appendix N. Most of the other changes are reorganization and rewording. Line numbers in our responses refer to the revised PDF. We did not add any new method, and none of the numbers reported in the submission has changed. The anonymous repository now includes the scripts and outputs for the new analyses.

AuthorsOpenReview ↗

Changes for each review comment

Line numbers refer to the revised PDF.

CommentChangeWhere
CA7g Q1/W2Evidence that logs without command history exist; statement that many recent systems do record setpoints§1, lines 40–79 (line 45)
CA7g W4, Q3; byyk W3Definitions of the label budget and schedule; even sampling presented as the most favorable schedule; campaigns and single windows for all strategies on all twelve configurations§1, lines 111–120; §3, lines 213–215; §5, lines 340–346; §6.2, lines 417–437; Table 2; App. L
CA7g Q4, W3, Q2A label defined as the issued command; shares of binding, floating and unmet setpoints; response one step after a setpoint change; inverse-model error compared with a monthly-median guess; no identifiability claim§3, line 213; §6.3, lines 440–464; §2, lines 184–188; App. M
CA7g W5Contribution stated as findings; comparison with LAWM shortened§1, lines 130–145; §2, lines 167–173
CA7g W1Huber-loss sentence removed; one name per strategy; plain wording; caveats moved to the limitations section§4; §5, lines 331–338; §7, lines 496–521
byyk (setup), n7gsNew Figure 1; separate sections for the problem formulation, method, experimental setup and resultsFigure 1 (page 2); §3–§6
byyk W1Comparison with building-control benchmarks and datasets§2, lines 190–197; App. N
byyk W2/W4Results organized around five questions, each with a takeaway§6.1–§6.5, lines 355–491
byyk (budget allocation)Decision rule for collecting and using the labels§8, lines 529–537

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.