- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates performance degradation in agentic cinematic subtext detection. It introduces LitSubText, a dataset of 2,606 dialogue instances from 132 film screenplays, with machine-generated annotations for communicative intent and thematic interpretation. The authors compare prompt-only configurations with a task-specific ReAct system, NarrAgent, and introduce RA-NarrAgent, which adds confidence-gated retrieval, a bidirectional critic, and an interpretation depth controller to discourage unsupported interpretations.
The experiments report that prompt-only configurations outperform the evaluated agent configurations. The proposed restraint mechanisms do not close this gap. Trajectory analysis shows that the full system never emits a SIMPLIFY signal, while controlled candidate probes suggest that the critic responds more strongly to label correctness than to unsupported elaboration. The paper presents these findings as evidence of configuration-dependent agentic overthinking.
Critical strengths
-
This paper investigates performance degradation in agentic cinematic subtext detection. It introduces LitSubText, a dataset of 2,606 dialogue instances from 132 film screenplays, with machine-generated annotations for communicative intent and thematic interpretation. The authors compare prompt-only configurations with a task-specific ReAct system, NarrAgent, and introduce RA-NarrAgent, which adds confidence-gated retrieval, a bidirectional critic, and an interpretation depth controller to discourage unsupported interpretations.
-
The experiments report that prompt-only configurations outperform the evaluated agent configurations. The proposed restraint mechanisms do not close this gap. Trajectory analysis shows that the full system never emits a SIMPLIFY signal, while controlled candidate probes suggest that the critic responds more strongly to label correctness than to unsupported elaboration. The paper presents these findings as evidence of configuration-dependent agentic overthinking.
Critical weaknesses
-
The experiments establish that the authors' agent configurations perform worse than their prompt-only baselines, but do not sufficiently rule out deficiencies in system design, controller training, critic feedback, or implementation as the primary explanation. Several observations reinforce this concern:
- The central comparison is confounded. The abstract and Section 5.2 describe the theme macro-F1 decline from 27.3% to 13.0% as resulting from switching on the loop while keeping the prompt fixed. However, Table 1 and Appendix C.2 identify these configurations as L3 without NFP and NarrAgent with NFP; prompt-only L3+NFP obtains 4.6%. Decoding temperature and controller adaptation also differ across relevant configurations. These results do not isolate the effect of additional deliberation.
- Controller SFT uses fixed-template trajectories without variation in retrieval decisions or stopping behavior (Appendix C). This leaves open whether the controller has learned an adequate action-selection policy for the task being studied.
- The full restraint system emits no SIMPLIFY signals, and most VERIFY actions do not yield a parsed direction. This raises questions about the online operation of the proposed feedback mechanism before its failure can be interpreted as evidence about restraint or overthinking.
- The cross-model experiment uses an unadapted Llama controller, with NarrAgent failing to terminate on 60.8% of instances. This is not a convincing independent replication of the proposed explanation, since inadequate controller behavior remains a direct alternative explanation. The component ablations and candidate probes provide useful diagnostics, but do not resolve these alternatives. Although the paper acknowledges several limitations and qualifies the degradation as configuration-dependent, those qualifications do not establish why the configurations fail. I am not asking for a universal claim across all agents; even a single-system study could be informative if it isolated the mechanism convincingly. Here, the evidence supports a narrower finding about these implementations, while the broader scientific interpretation as agentic overthinking remains insufficiently substantiated.
-
The dataset is selected and labeled in the same LLM pass, with null examples collected under a different instruction and a per-screenplay quota. Human agreement on the fine-grained labels is low: Appendix B reports human–human Cohen's kappa of approximately 0.181 for intent and 0.145 for theme. Moreover, retrieved context may support interpretations unavailable to the original local annotator. Evaluating every system against the same reference is useful for comparison, but does not ensure that reference bias affects all systems equally. Without an independent assessment of whether generated claims are supported by the evidence available to each system, reduced classification agreement cannot establish increased over-interpretation. This limits both the interpretation of the degradation and the proposed account of its causes.
Questions for authors
-
What evidence distinguishes the reported degradation from deficiencies of the particular agent implementation? In particular, after controlling the prompt, decoding settings, controller adaptation, and feedback functionality, does additional deliberation still cause degradation? Please reconcile the claimed fixed-prompt 27.3%→13.0% comparison with Tables 1 and 6, which indicate different NFP settings. This is my main question because it determines whether the paper establishes more than an unsuccessful system configuration.
-
Of 487 VERIFY actions, only 70 yield a parsed direction, and none yields SIMPLIFY. What happens in the remaining cases? Is this due to parsing, missing inputs, or another execution condition? How do the reported 95.2% held-out direction accuracy and the online behavior relate when evaluated on actual agent-generated candidates? Please distinguish the absence of an appropriate critic signal from failure to extract or apply it.
-
Is there an independently assessed subset showing that answers changed by the loop contain unsupported claims, rather than alternative interpretations or interpretations supported by retrieved context? In particular, does the system ordering persist against independent judgments of claim support? Such evidence would materially affect my assessment of the proposed explanation.
-
Given that the Llama controller is unadapted and NarrAgent fails to terminate on 60.8% of instances, what evidence supports interpreting its low scores as replication of overthinking rather than inadequate controller training? Does the degradation remain among valid, completed trajectories? Please distinguish evidence currently available from additional experiments proposed for future work.