acceptodds
Under review as a conference paper at ICLR 2027

Safe Executions Can Hide Policy Failure: A Proposal-Level Audit of Hybrid Control in Synthetic Interviews

Abstract

Safe executions can conceal unsafe proposals or a policy that avoids useful actions. We audit these failure modes in a synthetic structured-interview benchmark using joint proposal, execution, intervention and utility traces. A common hybrid-action interface supports TD3-style and PPO controllers with memory, reliability conditioning, nuisance consistency and proposal-cost optimization. Across 60 fitted policies and 23,520 evaluation episodes, both full controllers avoid follow-ups in the nominal panel, while a transparent rule achieves higher utility. Component-stack effects reverse direction between algorithms. An auxiliary control isolates exploration-induced constraint pressure: training exploration raises an always-proceed controller's mean discounted proposal cost to 0.696, above its budget of 0.5, despite zero deployed proposal cost. Released checkpoints, turn traces and executable audits connect every outcome table to its source, with explicit denominators and paired subject-level uncertainty. The findings motivate evaluating executed safety jointly with proposal safety and useful activity, and distinguishing exploratory behavior costs from target-policy costs. The evidence concerns a controlled synthetic setting; real-audio transfer and clinical effectiveness remain untested.

Primary area:
reinforcement learning
Keywords:
Safe reinforcement learning, hybrid control, proposal-level safety, safety shielding, dialog policy, auditability, structured interviews, synthetic benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_2WDBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces InterviewLite, a synthetic structured-interview environment for auditing hybrid RL controllers under a proposal-execution distinction. It evaluates TD3-style and PPO controllers with optional memory, reliability conditioning, nuisance consistency, and proposal-cost optimization, and argues that executed safety alone can hide either shield-corrected unsafe proposals or degenerate no-follow-up behavior. The empirical study reports 60 trained policies and compares proposal safety, executed safety, intervention, return, overlap, and useful clarification outcomes. The paper’s main conclusion is deliberately scoped: in this synthetic setting, joint reporting of proposals, executions, and utility exposes failures that would be hidden by executed safety alone.

Critical strengths
  1. The proposal-versus-execution accounting is a useful evaluation contribution. Equation 1 defines cost on unsafe proposals, including proposals blocked by the shield, while Section 5.1 clearly explains why executed unsafe rate alone cannot identify whether a policy is intrinsically safe or merely corrected by intervention.
  2. The hybrid-action formulation is more explicit than is common in small applied RL studies. In particular, Equations 3 to 6 specify how the continuous action and seven-way categorical action enter the TD3-style critic and actor update. Enumerating exact one-hot categories in Equation 6 is preferable to passing a fractional categorical representation into a critic that is only queried at discrete actions. The authors also appropriately acknowledge the residual surrogate-to-mode mismatch.
  3. The experimental accounting is careful. Section 6 explains that repeated evaluation seeds do not create additional independent subjects, aggregates at the subject level, preserves within-subject pairing across methods and seeds, and avoids treating dependent turns as independent samples. Equation 10 makes the paired estimand explicit. The paper also correctly separates pooled event rates from subject-averaged rates and explains why their denominators differ when policies induce different episode lengths.
  4. The main tables communicate the central diagnostic well.
Critical weaknesses
  1. The scientific contribution is constrained by a highly transparent, largely hand-specified fixture, and the paper does not demonstrate that its audit protocol changes conclusions outside this particular construction.
  2. The main paper does not provide enough empirical evidence for the broad “60 fitted policies” study it describes.
  3. The claimed exploration-pressure diagnosis is not fully specified or quantitatively established in the main paper.
  4. The TD3 and PPO optimization descriptions remain underspecified in ways that matter for the comparison.
Questions for authors
  1. For the TD3-style controller, what exact behavior-policy exploration is used during data collection for the continuous and categorical action components? Please distinguish this from the target smoothing noise in Equation 3. Is the stated 20% exploratory rate applied to all TD3 training runs, only to the always-proceed control, or both?
  2. Can the authors provide a compact summary of all six component settings across the nominal, dropout, and slow conditions? At minimum, this should include return, useful probes, proposal unsafe rate, executed unsafe rate, and total probes. The current Tables 1 to 3 only establish the full-versus-bare contrast.
  3. Would the authors consider evaluating the audit protocol on at least one environment beyond InterviewLite, with a separately declared proposal-level safety predicate?
Reviewer_dMLfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission examines a reinforcement learning interviewer in a synthetic structured-screening environment. The authors argue that looking only at what actually gets executed can hide two problems: the shield may be doing all the safety work, or the policy may simply avoid doing anything useful. They train 60 TD3-style and PPO controllers (2 algorithms, 6 component settings, 5 seeds) and evaluate them over 23,520 episodes with the shield on and off. Their findings: (a) both full controllers learn to never ask follow-ups, so they get no useful clarifications, and a simple hand-written rule and a "never follow up" policy both beat them; (b) adding the memory, reliability and consistency components helps PPO and hurts TD3; (c) a control experiment shows that even an "always proceed" policy gets a training cost of 0.696, above the 0.5 budget, because of random exploration, so the cost penalty is partly charging for exploration. They release checkpoints and traces and say the results do not transfer to real audio or clinical use.

Critical strengths

In this paper, the exploration-only control is a clear and verifiable demonstration that a Lagrangian cost update can be driven by exploration noise rather than by the behavior of the deployed policy. This is a consequential observation because it shows how the constraint signal and the evaluated policy can diverge.

Critical weaknesses

The empirical support is narrow relative to the claims in the abstract and conclusion. All results derive from a single hand-built simulator with a single hand-specified reward and five seeds per configuration. The reported reversal of component effects across algorithms may reflect untuned hyperparameters rather than a genuine interaction. The exploration control is close to true by construction, since 20% random actions will exceed a 0.5 cost budget. Table 3 reports an identical Holm-adjusted p-value of 0.013 for every contrast, which appears to reflect the resolution of the resampling procedure rather than the strength of evidence. Presentation is also weakened by extensive related-work discussion of components that are not evaluated (VAP, LLM baselines, real audio) and a motivating figure unrelated to the experiments.

Questions for authors
  1. Is the Holm p of 0.013 in every row of Table 3 just the smallest value your number of resampling draws can produce? How many draws did you use?
  2. Would the main finding work if either algorithm were tuned until it beat the no-follow-up rule? If not, the paper describes an under-tuned baseline.
Reviewer_uKsnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper audits proposal safety, executed safety, and utility in a synthetic interviewing task. Experiments with TD3-style and PPO controllers show that safe executions can hide unsafe proposals or complete avoidance of follow-ups. Component ablations and an exploration-only control examine how these behaviors relate to the training setup.

Critical strengths
  1. Logging proposals alongside executed actions makes it clear when safety comes from the shield rather than the policy.

  2. The rule and no-follow-up baselines are well chosen. They show that safe, useful behavior is possible and that zero violations alone do not imply success.

Critical weaknesses
  1. The main lesson largely follows from the task design: the shield blocks unsafe actions, while always proceeding incurs no cost. The audit illustrates this clearly, but offers limited additional insight.

  2. Both no-cost controllers still make zero probes in all three conditions. The paper documents this failure without establishing why the learners fail to find a safe, useful strategy.

  3. Evaluation uses argmax for the learned categorical policies. It remains unclear whether useful probes have negligible probability or simply never become the most likely choice.

  4. The nominal return reversal is mainly a timing effect, and both full models underperform their bare counterparts under slow conditions. The paper does not explain what drives these differences.

Questions for authors
  1. Do the full and no-cost policies assign meaningful probability to useful probes at eligible states? How do their results change when actions are sampled instead of selected by argmax?

  2. After separating timing and probe rewards, what explains the change in the full-versus-bare comparison between nominal and slow conditions?

Reviewer_6mkzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how safety evaluations can overlook failures in adaptive interviewing policies. In the synthetic InterviewLite environment, experiments with modified TD3 and PPO show that safe executions can result from external shielding or avoidance of follow-up questions, while training exploration can incur costs absent from deployed behavior. The paper advocates jointly evaluating proposal safety, interventions, and utility.

Critical strengths
  • Useful evaluation perspective. The paper highlights that safe execution alone does not establish a useful interviewing policy. Distinguishing shield correction from action avoidance helps interpret safety results alongside meaningful task engagement.

  • Informative comparisons. The algorithm and component comparisons show that the same component stack can affect TD3 and PPO differently. The auxiliary exploration control also illustrates a mismatch between training-time costs and deployed-policy costs.

  • Transparent reporting. Explicit denominators, paired subject-level evaluation, and trace-based result reconstruction make the synthetic findings inspectable. The paper clearly states the limits of their applicability to real interviews.

Critical weaknesses
  • Limited evaluation coverage. The experiments are confined to an author-designed synthetic environment with predefined rewards and controller configurations. Evaluation on established benchmarks or representative interviewing systems would help clarify how broadly the findings apply.
  • Missing domain comparisons. The experiments use modified TD3 and PPO alongside simple rules, but include no specialized interviewing or turn-taking methods. They therefore do not establish whether the observed failures affect approaches actually developed for the motivating application.
  • Limited diagnostic contribution. The paper documents avoidance but does not identify whether it results from reward scaling, exploration, insufficient training, or deterministic action selection. The central observations—that shielding can hide unsafe proposals and inactivity can satisfy safety constraints—largely follow from the environment design. The experiments add limited explanation beyond demonstrating these possibilities.
Questions for authors
  1. Do these findings also hold on existing benchmarks and with specialized methods for interviewing or turn-taking?
  2. What concrete guidance does this analysis offer for designing better interviewing policies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.