acceptodds
Under review as a conference paper at ICLR 2027

Bridging Efficiency and Morality: Ethical LTL Construction for Faster Convergence in Q-Learning‘s Ethical Dilemmas

Abstract

AI agents make decisions before executing actions, and we expect AI systems to make these decisions independently in large-scale autonomous deployments. Decisions made by AI agents are vital for preventing harm to humans and prioritizing human lives, both of which are considered moral objectives. Thus, the objective of building a general-purpose robot is to establish a mechanism for monitoring ethical decision-making by studying the decision-making process in partially observable Markov decision process (POMDP) environments. The challenge is how to generate an ethical linear temporal logic specification for model checking to ensure that the decisions made by AI are moral. Existing approaches use rule-based metrics or utility functions to constrain AI decision-making, but they are easily compromised because they lack a reasoning system. Our proposed approach is designed from a psychological perspective and trains the agent to learn a model-checking logic system without prior knowledge. We evaluate the approach on a modified burning-room problem and demonstrate its robustness and flexibility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI ethics, linear temporal logic, reinforcement learning, model checking, ethical decision-making, partially observable Markov decision processes

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JsPxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors claim that the paper presents a solution for applying "ethical temporal logic" to a RL game based on Q-learning. In actual fact, it is very difficult to understand how the temporal logic techniques are actually applied to the problem taken into consideration (the authors reported only one equation without explaining it) and why the proposed solution can be considered "ethical" and lead to efficient training.

Critical strengths
  • The area of using RL with "shields" is an interesting and active one.
Critical weaknesses
  • The definition of the problem is unclear. For example, the authors use fixed values of parameters that are not justified and the actual game itself is essentially not described in sufficient detail. The formal definition of the constraints is again very hard to understand. The authors report Equation 2, but it is not explained in terms of its structure. The equation also includes symbols that are not defined. The actual use itself of this equation is not discussed. The authors talk about "temporal logic", but thel definition of the "temporal logic" problem is not discussed in the paper.

  • Many parts of the manuscript are very hard to understand, somehow disconnected from the rest of the paper. The authors keep talking about ethical temporal logic, but this concept is not really defined. What could be considered ethical in the model the authors describe? The intuition at the basis of the proposed approach is very unclear: besides the aspect of ethics, why should the proposed approach speed up the learning in that specific case? Essentially, the authors present one case study: it is difficult to understand how the proposed model could generalise to others.

  • In Table 1, given the fact that the confidence intervals (?) are overlapping, there is no clear statistical advantage obtained by the proposed method. The interpretation of this table does not appear to be correct.

Minor comment:

  • The illustration of the work-flow is also very unclear and somehow disconnected from the description of the approach described in the text.
Questions for authors

The reviewer has a negative view of this paper. Unfortunately, the major concerns expressed above cannot be addressed easily.

Reviewer_9RhEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the ethical decision making problem in a Q-learning framework integrating LTL. The method is illustrated via a burning room example and also tested on it. The proposed method shows improvement over the vanilla Q-learning method.

Critical strengths
  • The method design looks reasonable.

  • Illustration using a concrete example helps with understanding the method.

  • Improvement is demonstrated over the standard Q-learning.

Critical weaknesses
  • Incorporating LTL for action planning or decision making, and even safety with reinforcement learning, has been well studied over recent years. A much wider and in-depth literature review is needed to explain the novelty of this work.

  • Using a single example for method development and experiments is not sufficient.

  • Overall, the paper reads like an initial attempt that's developing a baseline method by combining existing approaches and testing on a toy example. There isn't sufficient consideration in method novelty or generalizability and the required comprehensiveness of experimental evaluations.

  • Some statements are not really correct, for example, in the abstract: "the objective of building a general-purpose robot is to establish a mechanism for monitoring ethical decision-making by studying the decision-making processing partially observable Markov decision process (POMDP) environments."

Questions for authors

See above.

Reviewer_kyikOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the integration of ethical constraints into Q-learning in a partially observable environment. The authors consider a simplified rescue scenario in which an agent must balance the collection of rewards against an ethical obligation to rescue a human. The proposed approach combines an LTL specification of the ethical obligation, a finite-state progress monitor incorporated into the Q-learning state representation, reward shaping through additional bonuses and penalties, and a pre-emptive shield that filters actions considered unsafe.

The approach is evaluated against standard Q-learning and penalty-based Q-learning. The experimental results indicate that the proposed method achieves higher average returns, fewer ethical violations, and higher rescue rates. It also reaches a predefined performance threshold in fewer training episodes. The authors additionally investigate the impact of observation noise on performance.

Critical strengths

The paper addresses an interesting and relevant problem at the intersection of reinforcement learning, temporal logic, and ethical decision-making. Incorporating explicit ethical constraints into learning agents is an important research direction, particularly when reward maximization alone may lead to undesirable behavior.

The proposed architecture is conceptually simple and combines several established techniques including state augmentation, reward shaping, and action shielding, in a coherent experimental setting. The paper is clearly written, and the presentation of the rescue scenario makes the underlying motivation accessible.

The experimental evaluation provides evidence of improvements over the two considered baselines in terms of average return, ethical violation rate, and rescue success. The additional experiments with observation noise are also useful for assessing the robustness of the approach under imperfect perception.

Critical weaknesses
  1. Unclear role of LTL. My main concern is that the paper does not explain how the LTL formula in Equation (2) is actually used in the proposed algorithm. The authors introduce a five-state progress monitor but provide neither a formal construction from the LTL formula nor its transition function. Moreover, the shielding mechanism appears to rely on manually specified action restrictions rather than on the temporal specification. Consequently, it is unclear whether LTL plays an essential operational role in the approach.

  2. Limited novelty and formal justification.The proposed approach combines established techniques, namely state augmentation, reward shaping, and shielding. Its contribution beyond existing work is not sufficiently clarified.

  3. Insufficient experimental analysis. The absence of ablation experiments makes it difficult to determine which components account for the reported improvements. The reward-shaping bonuses and action restrictions alone might explain much of the observed performance gain. Furthermore, the experiments are limited to a small environment, and the claimed faster convergence refers to reaching an empirical performance threshold rather than a theoretical convergence result.

Questions for authors
  1. How exactly is the LTL formula translated into the progress monitor? Could the authors provide its transition function and clarify whether the monitor is automatically generated or manually constructed?

  2. Is the shielding mechanism systematically derived from the LTL specification, or are its action restrictions manually defined? What formal guarantees, if any, does it provide?

  3. How are the reward-shaping bonuses determined? Are they derived from the LTL specification or manually selected?

  4. Could the authors provide ablation experiments separating the effects of the progress monitor, reward shaping, and shielding?

  5. What is the main methodological novelty compared with existing work on LTL-guided reinforcement learning and shielding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.