acceptodds
Withdrawn from ICLR 2027

FlowDVFS: Outperforming Generative Replay for Few-Shot DVFS Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-shot reinforcement learning for dynamic voltage and frequency scaling (DVFS) must find, from a few dozen real transitions, the clock giving the highest stable frame rate without throttling. We propose FlowDVFS, a flow-matching replay generator for a few-shot Double DQN: a physics anchor backed up at every clock, plus a flow residual. We judge the generator by a learning curve over 8 to 32 tuples on a throughput reward whose thermal target never binds in the replayed runs. On held-out traces of a Jetson Orin NX and TX2, at MBPO's twenty updates per real transition, FlowDVFS needs fewer real tuples than SynthER, PGR (one update) and MBPO and is above all three at every budget, tested at each against MBPO. Neither replay method cuts the learner's real budget; FlowDVFS does, its anchor's argmax with no learner matching it on development, while the top clock, best on every run, stays above every learned method.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
reinforcement learning, few shot learning, flow matching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7d7DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method called FlowDVFS, which combines physical anchors, conditional flow matching, and dual DQN learners to achieve small-sample DVFS control. The evaluation results show that in the trajectory-driven experiments on two Jetson boards, this method outperforms the generative replay and the improved MBPO baseline model. However, the paper's own attribution experiments indicate that these performance gains mainly come from the physical anchors, while the flow matching or the learned strategies do not bring measurable additional benefits.

Critical strengths

The application is practically relevant. The paper includes extensive ablations and held-out trace evaluations, and transparently reports negative results, including the lack of measurable gains from the flow residual and the loss of the sample-efficiency advantage under a binding thermal constraint.

Critical weaknesses
  1. The paper does not establish the added value of flow matching or reinforcement learning. Table 1 and Appendices A.3–A.5 show that the physics anchor alone performs comparably to the full method. On the development split, directly maximizing the anchor’s predicted reward without a learner achieves higher mean reward at every evaluated budget than any learned method. Moreover, the zero discount factor reduces the learning target to immediate reward prediction. These results attribute the gains primarily to the physical prior and leave the methodological contribution of the proposed framework unsubstantiated.
  2. The main experiments do not adequately test thermally constrained dynamic control. The thermal constraint never binds, and the highest fixed clock achieves the best reward on every recorded run. Under the binding constraints evaluated in Appendix A.3, FlowDVFS no longer improves the mean sample-budget metric over the control and incurs more constraint violations. The results therefore do not establish that the headline sample-efficiency advantage extends to settings requiring a meaningful trade-off between throughput and temperature.
Questions for authors
  1. Can you identify an evaluated condition in which the full FlowDVFS method provides a clear benefit over directly maximizing the physics anchor’s predicted reward? If no such condition is established, what contribution should be attributed specifically to flow matching and policy learning?
  2. In Appendix A.3, the sample-efficiency advantage disappears while constraint violations increase. Do the available results demonstrate any performance or sample-efficiency advantage when methods are compared at similar constraint-violation rates? This would materially affect my assessment of the method’s effectiveness for thermally constrained control.
Reviewer_jtzFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. FlowDVFS studies few-shot control of CPU frequency on Jetson Orin NX and TX2. It fits a physical response model for frame rate, power, and temperature, then learns a conditional flow-matching residual around that anchor.
  2. The generator supplies synthetic successors at every clock for observed states, augmenting a DQN learner. Evaluation measures the number of real tuples needed to match the unaugmented learner's full-budget reward, using recorded-trace simulation and comparisons with SynthER, PGR, and MBPO.
  3. The study includes anchor-only, no-learner, and fixed-clock controls, a held-out evaluation, and separate induced-throttling runs. The reported sample-efficiency gains are attributed primarily to the physical prior; the experiments do not resolve an additional benefit from the flow residual.
Critical strengths
  1. The simple controls are the most informative part of the paper. The anchor-only model and direct anchor argmax show what the physical prior already achieves. Reporting that the flow and learner add little at the available resolution is useful, even though it weakens the case for the proposed architecture.
  2. The learning-curve evaluation asks a concrete question about real-data requirements rather than judging synthetic samples only by fidelity. The held-out comparison covers 148 Orin NX and 82 TX2 runs, and paired contrasts reduce variation caused by different input traces. Within this setting, the anchored method consistently improves on the tested replay baselines.
Critical weaknesses
  1. The experiments do not establish a benefit from the main proposed flow component. The anchor-only model largely matches FlowDVFS, while direct anchor argmax already performs well without a learner. The fixed top clock is best on every held-out run. The observed advantage over generative replay therefore supports the physical prior much more directly than it supports a new flow-based replay method.
  2. The main evaluation does not exercise the thermal trade-off motivating the application. Thermal violations are zero in every held-out learning-curve cell, and the data come from one workload on two individual boards. Separate induced-throttling runs show that adverse events can occur, but do not compare controllers when avoiding those events affects reward. As a result, the central control problem remains insufficiently tested.
  3. Much of the theoretical discussion concerns a discounted learner or independent samples from the true dynamics, whereas the deployed learner uses zero discount and synthetic samples from a fitted model. The manuscript acknowledges these differences, but does not provide a bound on the anchor or residual model's error that would connect the discussion to its method. The theory consequently offers little explanation of when the proposed generator should help or fail.
Questions for authors
  1. Is there a controlled evaluation with a binding thermal constraint where FlowDVFS improves on both anchor-only control and a fixed-clock policy?
  2. In a closed-loop hardware comparison at the same real-transition budget, what benefit can be attributed specifically to the flow residual rather than the physical anchor?
  3. Which theoretical conclusion applies to the implemented zero-discount learner with model-generated samples? Can the relevant model-bias term be quantified, rather than relying on independent draws from the true environment?
Reviewer_g58fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FlowDVFS, a physics-anchored flow-matching generator for few-shot reinforcement learning in dynamic voltage and frequency scaling (DVFS). It combines a physics-based transition model with a learned flow residual to generate synthetic transitions for Double DQN training.

Experiments on Jetson Orin NX and TX2 show better sample efficiency than SynthER, PGR, and MBPO in a trace-driven evaluation. However, the ablations show that the physics anchor alone achieves essentially the same performance as FlowDVFS, raising questions about the actual contribution of flow matching.

Critical strengths
  1. Careful empirical evaluation. The paper evaluates learning curves across multiple data budgets on two hardware platforms, includes held-out runs and statistical tests, and compares against several relevant baselines. I also appreciate that the authors explicitly report negative ablation results rather than hiding them.

  2. Practical motivation and useful findings. Few-shot control on embedded hardware is an interesting problem. The results suggest that simple physics-based models can be more effective than complex generative replay in highly data-constrained settings.

Critical weaknesses
  1. The main contribution of flow matching is not demonstrated.

The physics anchor alone matches or slightly outperforms FlowDVFS on the held-out runs (Appendix A.4). Additional ablations also show little benefit from flow-related components or tuning. This suggests that nearly all the improvement comes from the physics prior and counterfactual action coverage, rather than the proposed flow-matching generator.

The novelty is therefore limited, particularly given existing physics-informed Dyna-style methods (El Asri et al., 2024) and earlier work applying flow matching and bootstrapping to few-shot DVFS (arXiv:2409.14178).

  1. The main evaluation does not adequately validate the claimed control problem.

The thermal constraint never becomes active in the main learning-curve experiments, and the highest fixed CPU clock is optimal across all evaluated runs. In fact, simply holding that clock achieves higher reward than the learned methods, without any training.

More importantly, when the authors introduce binding thermal constraints (Appendix A.3), the claimed sample-budget advantage disappears, and the anchored methods show higher thermal-violation rates than the control.

This makes it difficult to conclude that FlowDVFS offers a meaningful advantage for actual thermally constrained DVFS control. The use of a zero discount factor also reduces the deployed learner to one-step reward estimation, while the theoretical regret discussion largely concerns multi-step discounted RL.

Questions for authors

What is the actual benefit of flow matching? Since the physics anchor alone performs just as well, can the authors identify a setting where the learned flow residual provides a clear and statistically meaningful improvement?

Does the method work when thermal constraints really matter? Can the authors explain why the sample-efficiency advantage disappears under binding constraints? A direct comparison with simple fixed-clock, physics-based, or contextual-bandit controllers under such conditions would help establish the practical value of the method.

How does the theoretical analysis apply to the deployed learner? With γ = 0, the Bellman target becomes the immediate reward, and the difference between Q* and Qπ at the same state-action pair is zero. Also, synthetic transitions generated from a fitted model are not independent samples from the true environment. It would help to clarify what theoretical conclusions remain valid for the actual algorithm.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.