- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether an initial constraint remains decodable as dialogue adherence declines, and how two steering methods interact over multiple turns. It constructs 230 sessions of 20 turns covering 23 constraint types, evaluates four Qwen and Llama models, and trains session-disjoint linear probes on states extracted before each answer.
Critical strengths
-
The primary probe avoids direct contamination from the current answer: The extraction excludes that answer, and PCA and classifier fitting use session-disjoint training folds (Section 4.2; Appendix W). This supports the narrow finding that constraint-type information is available before many checker failures, while the paper correctly avoids equating decodability with functional use.
-
The intervention analysis examines both benefits and damage: The study reports the complete integer strength grid, random and wrong-constraint controls, and paired whole-session intervals (Section 6; Appendix I). Distinguishing subadditivity from marginal benefit, and reporting lost as well as gained successes, makes the comparison useful for assessing whether separately helpful controls combine successfully.
Critical weaknesses
-
The task and labels do not consistently measure persistent adherence. Some prompts refer only to "the response," although every later answer is scored against the initial target (Section 4.1). Table 4 also includes constrained-choice prompts without choices scored using a 50-word limit, and JSON checks that accept arbitrary values or fenced blocks despite requests for an object. These problems affect both intervention effects and failed-turn probe subsets, and no corrected-scoring sensitivity analysis is provided (Appendix V).
-
The interventions lack independent evaluation. Qwen’s layers and head counts were selected by maximizing checker performance on the same 230 sessions later used for evaluation (Appendix F). Session-specific steering targets and attention calibration also use these sessions' recorded trajectories, so the bootstrap intervals do not establish that the selected controls transfer to fresh conversations.
-
Pervasive truncation limits the meaning of the gains. Between 62.35% and 80.20% of baseline answers reach the 256-token cap, while steering substantially changes stopping rates (Section 6.5). Without a larger-budget comparison and an answer-quality assessment, gains on JSON, closing quotations, and ending phrases may partly reflect whether responses finish, leaving their practical value unresolved.
Questions for authors
- How many instruction variants explicitly require adherence on every turn, and do the main findings persist on that subset with faithful checks and malformed constrained-choice cases excluded?
- Do the intervention gains and joint-interaction patterns replicate on untouched sessions, with steering sites and calibration chosen using separate data?
- Do these findings persist with a response budget that allows most answers to finish, and do the rescued answers still answer the content questions correctly?