acceptodds
Withdrawn from ICLR 2027

Efficient Charging Strategies for Electric Trucks using Actor-Critic Reinforcement Learning with Tree Ensembles

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-haul electric trucking requires charging strategies that balance delivery efficiency against battery health, under uncertainties such as stochastic energy consumption. We formulate this as a profit-maximizing sequential decision problem: At each charging station, an agent chooses how long to charge and at what power, trading delivery revenue against electricity cost, driver wages, and battery degradation. The environment is inherently non-stationary since the agent's own decisions determine how battery capacity evolves over time. Addressing this problem well calls for methods that remain effective under drifting dynamics and can run on constrained, embedded, in-vehicle compute. To address this challenge, we propose tree ensembles combined with reinforcement learning. Tree ensembles excel on tabular low-dimensional data typical of such control problems, but remain largely unexplored for actor-critic reinforcement learning. The only existing method, which grows ensembles without bound during training, incurs continuously increasing memory and compute costs. As an alternative, in this paper we introduce Tree Ensemble Reinforcement Learning (TERL), which instead fixes the ensemble size. To update the model, TERL periodically rebuilds trees with off-policy reinforcement learning, and refits leaf values with on-policy methods between rebuilds. This requires no incremental tree-growing interface, while still letting the ensemble continuously learn and adapt to non-stationary dynamics. We instantiate TERL with XGBoost and demonstrate its competitiveness against existing methods. We evaluate both on standard benchmark problems and our electric-truck charging environment, offering a realistic testbed for EV charging optimization and a general, resource-efficient framework for tree ensemble reinforcement learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
reinforcement learning, actor-critic methods, gradient boosting, tree ensembles, sequential decision making, electric vehicle charging

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mp3NOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies charging control for electric trucks and introduces TERL, an actor-critic framework using tree ensembles of fixed size. It alternates periodic rebuilding through advantage-weighted regression with on-policy leaf updates. Experiments on Pendulum and a simulated charging corridor compare TERL with neural and GBRL agents. TERL achieves higher charging profit than the reported GBRL baselines and lower inference latency, although the best neural policy achieves higher profit.

Critical strengths
  1. Fixing the ensemble size addresses the growing model cost of GBRL. Periodic rebuilding allows the tree structure to change while leaf updates adapt the policy between rebuilds. The reported inference results support the usefulness of this design.

  2. The charging task combines battery degradation, uncertain energy consumption, electricity prices, and driver-rest constraints. Evaluation across seasons and battery-health levels provides a useful setting for studying these competing objectives.

Critical weaknesses
  1. The experiments do not separate the effects of rebuilding, leaf updates, and replay recency. Similar results for TERL-A2C and TERL-PPO do not establish which component produces the improvement.

  2. The claimed non-stationarity needs clarification. Battery capacity and calendar information are already included in the state. Their evolution alone does not establish that the transition rules change over time.

  3. Battery degradation is included in the reward, and battery capacity persists across missions. However, the evaluation does not directly show battery lifetime or cumulative profit across missions. The handling of returns at mission boundaries and any additional replacement cost needs clarification.

  4. The supplied manuscript does not give numerical values for the degradation coefficients and several reward parameters, including driver wages, battery depreciation, and the failure penalty. These values are needed to reproduce the charging trade-off.

  5. The paper lacks necessary diagrams to provide readers with a clear and intuitive understanding of the research content and methodology process.

Questions for authors
  1. How does TERL compare with periodic AWR rebuilding without leaf updates, and with leaf updates without rebuilding, under comparable budgets?

  2. After accounting for battery capacity and calendar information, which part of the transition dynamics changes during training?

  3. Does the return stop at the end of each mission? Is battery replacement charged separately, or already covered by the depreciation term? Can you report battery-health trajectories and cumulative profit across missions?

  4. Can you provide numerical values for the degradation coefficients, driver wages, battery depreciation, and failure penalty? How sensitive are the results to the assumed degradation cost?

1 comment

We would like to express our gratitude to the reviewer for showing great interest in our work and providing relevant comments. We attempt to answer the reviewer's questions as follows: 1 - We are unfortunately not able to run the experiments the reviewer asks for within the time-frame for the rebuttal phase. 2 - The part of the transition dynamics that remains changing is the temperature, as it evolves based on real historical records from meteostat. 3,4 - The return does not stop at the completion of one mission, but continues for the indefinite future, and the cost of battery replacement is assumed to be covered by the depreciation term. Unfortunately, we are not able to reveal the specific numerical values used, due to the interest of the collaborating company that has provided the information.

Reviewer_xYqZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper has a double intended contribution (i) proposing a new environment intended to realistically model a EV truck delivery environment where the agent has to account for the need to charge and the potential long-term damage to battery when charging faster; (ii) a new RL method leveraging an ensemble of trees for RL value predictions.

The validation of the method is performed both in the new proposed domain and in a simple Pendulum environment, and shows some relativelly small advantage of the new method compared to "normal NN-based RL".

For contribution (i) I am not quite sure this is the right venue since an electrical engineering journal would have more appropriate reviewers to review how realistic the battery decay model is. To begin with, there is not a very clear explanation of how the problem modelled here couldn't be modelled in standard software such as GridLAB-D, or in many coordinated charging environments or papers in the literature. If the authors want to focus on contribution (i) for the paper the phrasing of the manuscript should be revised to make it clear why we need such environment in first place, then building empirical information to show that the environment is realistic. Maybe assuming that this is the case, the evaluation should bring other approaches to similar problems to show that RL is indeed a good solution, such as optimization, genetic algorithms, plain search, etc. To not deviate much from the current manuscript the best course of action would be showing the RL overperforms the other approaches but is unrealistic in some aspect (model size?), motivating then to develop the new RL approach.

If the focus in contribution (ii) then it's a bit hard to argue that the validations are done only in a silly Pendulum environment (which btw is completely out of place in the paper and should just be reviewed) and the proposed new environment. You would need to show that the proposed method is abl e to solve a contemporary RL environment such as google football, starcraft, or something that would require good function approximation. In general I am not very convinced in the current manuscrupt that the proposal could solve a hard environment since it looks so classical.

Critical strengths

If well-made, the proposed domain might be very interesting given the clear mapping to a real application of commercial impact.

Critical weaknesses

The main weakness of the paper was trying to focus on two different contributions and in the end not delivering either of them. In the end of the paper I don;t trust the proposed RL method could solve a contemporary domain better than the state of the art, and am not fully convinced the proposed environment is a realistic and well-modeled problem where RL would be a top solution.

Questions for authors

No specific question that would change my evaluation.

1 comment

We sincerely thank the reviewer for their thorough comments and thoughtful recommendations. Due to limited rebuttal time, we will not be able to update the manuscript with the suggested changes and additional experiments. However, we will consider them when resubmitting the work to another venue.

Reviewer_tpCGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work studies charging strategies for long-haul electric heavy-duty trucks and formulates a profit-maximizing MDP. It develops a Tree Ensemble RL approach to solve the problem, but with limited new design or comparisons.

Critical strengths

This work studies a practically motivated problem of electric-truck charging using actor-critic RL with tree ensembles.

Critical weaknesses
  1. This work focuses on the long-haul electric truck charging task but does not provide new modeling or identify unique research problems about long-haul electric truck. The model looks generic and oversimplified, like other electric vehicle charging, even without considering the charging impact on the grid or emission, often regarded as necessary problem setup.

  2. There is a large body of literature missing from the literature review or comparisons, mainly on symbolic or neuro-symbolic RL methods. The proposed Tree Ensemble RL approach is not novel and lacks necessary theoretical analysis. For example, the cross-tree leaf-update rule is heuristic, and the leaf aggregation as a least-squares projection overstates the method.

  3. The simulation environment is also very limited, lacking realism of long-haul electric truck, power/energy systems, system uncertainties involved.

Questions for authors

See the weaknesses.

2 comments

We thank the reviewer for their comments and honest opinion, but disagree that our proposed Tree Ensemble RL approach is not novel. The closest existing competitor of TERL is the GBRL framework. Despite the impressive performance and promising features of GBRL, the specific way in which it utilizes functional gradient descent leaves the technique with a handful of issues. In particular, due to the continuous generation of new trees, the ensemble grows indefinitely, which limits the application of the method to real-time online adaptation. Additionally, the unbounded ensemble size may lead to computational inefficiency and memory issues. For a future resubmission, along with further developing the electric truck charging task based on the suggestions given here, we will focus on more clearly highlighting the novel aspects of our TERL and their importance.

Thank the author(s) for the response.

The motivation provided relative to GBRL is reasonable. However, the response does not explain how TERL resolves this issue. Moreover, the response defers most issues to a future resubmission, so I maintain my score and would encourage the author(s) to address the weaknesses in the future submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.