- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TAIRL, a temporal-aware interactive reinforcement learning framework for multi-turn open-ended dialogue. The authors analyze human feedback in financial-domain dialogues and identify three temporal patterns: turn-point concentration, asymmetric feedback timing, and adaptive feedback frequency. Based on these observations, the paper introduces three mechanisms: temporally-guided dynamic weighting (TDW), which emphasizes feedback near dialogue state transitions; asymmetric learning rate (ALR), which assigns larger update coefficients to negative feedback; and adaptive experience replay (AER), which prioritizes experiences collected during low-performance periods or important state transitions. The proposed framework is integrated with several existing RL methods, including PPO, GRPO, REINFORCE, and TAMER. Experiments are conducted on four dialogue datasets covering financial, medical, and home-service scenarios. Using a feedback simulator trained on real human annotations and a held-out human evaluation set, the authors report improvements in task success, learning efficiency, training stability, and user experience.
Critical strengths
- Meaningful research motivation based on human feedback patterns. The paper investigates temporal characteristics of human feedback that may be overlooked when feedback is treated only as a scalar reward. The empirical analysis of feedback concentration, response delay, and frequency provides a useful perspective on improving learning from human interactions. This is important because feedback timing and dialogue state transitions may contain useful information beyond the numerical reward value, especially in multi-turn interactions.
- Modular framework with promising empirical results. TAIRL is designed to integrate with several existing policy optimization algorithms without replacing their underlying optimization frameworks. The proposed mechanisms are relatively lightweight and can potentially be applied to different interactive RL settings. The experiments report improvements across multiple algorithms, language models, and dialogue datasets. The inclusion of ablation studies and held-out human evaluations further strengthens the empirical investigation.
Critical weaknesses
- Limited evidence for the specific benefits of temporal modeling and the claimed synergy between components. Although the three mechanisms are motivated by temporal patterns in human feedback, their implementations primarily involve gradient weighting and prioritized experience replay. In particular, ALR assigns different update coefficients according to feedback polarity, but does not directly incorporate the observed feedback delay. Therefore, it is unclear whether its improvement comes from exploiting temporal characteristics or simply from assigning larger weights to negative feedback. Furthermore, TDW, ALR, and AER appear to be relatively independent optimization mechanisms. Although AER incorporates temporal weights from TDW, the current leave-one-out ablation studies do not fully establish whether their combination produces benefits beyond those of independently effective weighting strategies. Since exploiting temporal feedback patterns and combining the three mechanisms are central contributions of the paper, stronger evidence is needed to distinguish their specific contributions from conventional feedback re-weighting and experience prioritization.
- Insufficient clarity and consistency in the experimental methodology and reported results. Several experimental details make it difficult to fully assess the reliability of the reported improvements. First, Table 9 reports the same FP and LS values for GRPO+AER as Table 3 reports for the complete TAIRL framework. However, Table 3 also shows substantial performance degradation when TDW or ALR is removed. The relationship between these configurations requires clarification. Second, Section 4.1 describes simulator-based training over 2,000–3,000 interaction turns, while some learning-speed results in Table 1 exceed this range. The experimental protocol used to obtain these convergence measurements is unclear. Finally, Appendix F describes a reward model predicting feedback from state-action pairs, while Appendix F.2 reports a time-shuffling experiment intended to demonstrate the importance of temporal structure. The manuscript does not sufficiently explain how the temporal information is represented or how shuffling time indices affects the simulator. These issues are consequential because the experimental evidence is central to the paper's claims regarding learning efficiency and temporal feedback modeling.
Questions for authors
- Regarding the ALR mechanism and temporal modeling: Since ALR depends on feedback polarity rather than the actual feedback delay, can the authors clarify what additional benefit is obtained from modeling temporal feedback patterns compared with a fixed negative-feedback weighting baseline? Additionally, can the authors clarify whether ALR is implemented through optimizer learning rates, per-sample loss weighting, or reward/advantage scaling in PPO and GRPO?
- Regarding component contributions and experimental consistency: Does GRPO+AER in Table 9 use only the AER mechanism? If so, why does it achieve the same reported performance as full TAIRL? Could the authors clarify this configuration and provide component-wise comparisons that better distinguish the individual contributions and interactions of TDW, ALR, and AER?
- Regarding the feedback simulator and training protocol: How are feedback timing and temporal ordering represented in the reward simulator? In particular, how does permuting time indices affect simulator training when the state-action-feedback triples remain unchanged? Could the authors also explain how convergence-turn measurements exceeding the stated 2,000–3,000 training turns were obtained?
Response to Reviewer3
We thank the Reviewer for the rigorous examination. All revisions are in the revised manuscript.
W1: Temporal Modeling Evidence & Synergy
(1) ALR vs. Conventional Re-weighting: We clarify that while ALR itself does not directly encode continuous delay seconds, it models the asymmetric temporal strategy (instant correction vs. delayed confirmation). We acknowledge that ALR in isolation is algebraically equivalent to sign-based re-weighting (w_attn=1 → fixed 10× on negative samples). Its temporal awareness emerges only through coupling with TDW (Eq. 11): α(r)·w_attn(t)·∇logπ. Negative feedback inside vs. outside the switching window receives different magnitudes (α⁻×β_in vs. α⁻×β_out). This "knowing when to amplify" is the essence of the framework, distinguishing it from static cost-sensitive learning. We therefore frame TDW/ALR/AER as a unified temporal framework rather than three independent modules.
(2) Synergy evidence (Table 3): Conventional Static Weighting (fixed 10×, no TDW/AER) achieves only 73.20%, far below Full TAIRL (79.83%). Note that w/o TDW (Static 10×+AER) = 73.61% is close to Static Weighting (73.20%), confirming AER contributes minimally without TDW (+0.41pp); its value emerges through cross-stage transmission (TDW→AER via w_attn in Eq. 14), yielding +5.31pp when combined. vs. PER: GRPO+PER (TD-error) = 74.21% (Table 12).
Table 3 (updated). D3, GRPO base.
W2: Experimental Consistency
(1) (1) Table 9 data overlap: This was a copy-paste typo; the row contained Full TAIRL data mislabeled as "GRPO+AER." We have corrected this. As shown in the revised Table 12 below, the "AER-only" configuration (w/o TDW, w/o ALR) achieves 74.10%, which is comparable to GRPO+PER (74.21%). This confirms that AER alone acts as a standard prioritized replay. The true synergy emerges only in Full TAIRL (79.83%).
(2) LS > 3k turns: Text inconsistency corrected. The actual max budget is 5,000 turns (most TAIRL variants converge within 1.0–2.3k turns; some baselines require up to 4.7k turns). Section 4.1 is updated to reflect this exact protocol; all Table 1 data verified against logs.
(3) Time-shuffling ( Appendix. F): Permutation is applied to training pairs before RM training, corrupting the (s,a)→r supervised mapping. We acknowledge this introduces an RM-quality confound and report shuffled-RM sign accuracy = 81.2% (vs. 87.6% original). To isolate temporal structure from RM degradation, we compare the differential effect: GRPO (no w_attn/AER) is barely affected (−0.31pp), while TAIRL drops 7.63pp,proving TAIRL specifically exploits temporal alignment. Temporal info is implicit (chronological feedback order + state context, not explicit timestamps); permuting r_t destroys critical-window alignment.
Table 12 (corrected). D3 head-to-head.
Table 15 (time-shuffle control). D3.
Questions
Q1: Static Weighting baseline (73.20% vs 79.83%) proves temporal localization matters. ALR = per-sample loss weighting (not optimizer LR / reward scaling). Multiplier stopgrad(w_attn·α) applies to unmodified 𝓛ᵢ: 𝓛ᵢᴾᴾᴼ = min(ρᵢAᵢ, clip(ρᵢ,1−ε,1+ε)Aᵢ); 𝓛ᵢᴳᴿᴾᴼ = ρᵢÃᵢ. Not applied to ρᵢ or Aᵢ.
Q2: "GRPO+AER" was a typo (Full TAIRL data mislabeled). Table 12 (revised). AER isolation on D3.
Q3: (1) Temporal info = chronological order + context; no explicit timestamps. (2) Permuting r_t before RM training corrupts (s,a)→r mapping (shuffled-RM accuracy = 81.2%); differential GRPO vs. TAIRL effect isolates temporal-structure dependence. (3) Max budget = 5,000 turns (corrected in §4.1).
All revisions are in the revised PDF.