acceptodds
Withdrawn from ICLR 2027

Inverse-Scaling Signature in Agentic Cinematic Subtext Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic pipelines that combine retrieval, decomposition and critic-based refinement are widely assumed to improve as inference-time computation increases. On cinematic subtext detection, which requires pragmatic inference about what characters mean but do not say, the assumption fails. Using LitSubText, 2,606 dialogue instances from 132 film screenplays annotated with 8 intent and 9 theme classes, a single forward pass outperforms every agentic configuration tested. Holding the prompt and the tools fixed and varying only whether the ReAct loop runs, theme macro-F1 falls from 27.3 to 13.0 percent and theme accuracy from 34.2 to 11.7 percent, a significant gap under a screenplay-level cluster bootstrap. The loop breaks 89.0 percent of what the prompt-only configuration answered correctly while repairing 12.0 percent of what it got wrong, so it substitutes predictors rather than partly improving one. Three restraint mechanisms were added to test whether the loss can be recovered, namely Confidence-Gated Retrieval, a Bidirectional Critic that can recommend simplification as well as deepening, and an Interpretation Depth Controller that stops early on critic direction signals. They do not recover it. Instrumented trajectories show that the simplify direction, the only signal expressing restraint, fired zero times across 240 test instances and that the depth controller produced longer trajectories than the unrestrained loop, so two of the three never operated as specified. Probes that hold the input fixed and vary only the candidate locate the reason, because the critic separates wrong candidates from right ones far more sharply than over-detailed ones from concise ones. What it supplies is largely a correctness signal rather than a restraint signal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, inference-time compute scaling, inverse scaling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_hd6dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates performance degradation in agentic cinematic subtext detection. It introduces LitSubText, a dataset of 2,606 dialogue instances from 132 film screenplays, with machine-generated annotations for communicative intent and thematic interpretation. The authors compare prompt-only configurations with a task-specific ReAct system, NarrAgent, and introduce RA-NarrAgent, which adds confidence-gated retrieval, a bidirectional critic, and an interpretation depth controller to discourage unsupported interpretations.

The experiments report that prompt-only configurations outperform the evaluated agent configurations. The proposed restraint mechanisms do not close this gap. Trajectory analysis shows that the full system never emits a SIMPLIFY signal, while controlled candidate probes suggest that the critic responds more strongly to label correctness than to unsupported elaboration. The paper presents these findings as evidence of configuration-dependent agentic overthinking.

Critical strengths
  1. This paper investigates performance degradation in agentic cinematic subtext detection. It introduces LitSubText, a dataset of 2,606 dialogue instances from 132 film screenplays, with machine-generated annotations for communicative intent and thematic interpretation. The authors compare prompt-only configurations with a task-specific ReAct system, NarrAgent, and introduce RA-NarrAgent, which adds confidence-gated retrieval, a bidirectional critic, and an interpretation depth controller to discourage unsupported interpretations.

  2. The experiments report that prompt-only configurations outperform the evaluated agent configurations. The proposed restraint mechanisms do not close this gap. Trajectory analysis shows that the full system never emits a SIMPLIFY signal, while controlled candidate probes suggest that the critic responds more strongly to label correctness than to unsupported elaboration. The paper presents these findings as evidence of configuration-dependent agentic overthinking.

Critical weaknesses
  1. The experiments establish that the authors' agent configurations perform worse than their prompt-only baselines, but do not sufficiently rule out deficiencies in system design, controller training, critic feedback, or implementation as the primary explanation. Several observations reinforce this concern:

    • The central comparison is confounded. The abstract and Section 5.2 describe the theme macro-F1 decline from 27.3% to 13.0% as resulting from switching on the loop while keeping the prompt fixed. However, Table 1 and Appendix C.2 identify these configurations as L3 without NFP and NarrAgent with NFP; prompt-only L3+NFP obtains 4.6%. Decoding temperature and controller adaptation also differ across relevant configurations. These results do not isolate the effect of additional deliberation.
    • Controller SFT uses fixed-template trajectories without variation in retrieval decisions or stopping behavior (Appendix C). This leaves open whether the controller has learned an adequate action-selection policy for the task being studied.
    • The full restraint system emits no SIMPLIFY signals, and most VERIFY actions do not yield a parsed direction. This raises questions about the online operation of the proposed feedback mechanism before its failure can be interpreted as evidence about restraint or overthinking.
    • The cross-model experiment uses an unadapted Llama controller, with NarrAgent failing to terminate on 60.8% of instances. This is not a convincing independent replication of the proposed explanation, since inadequate controller behavior remains a direct alternative explanation. The component ablations and candidate probes provide useful diagnostics, but do not resolve these alternatives. Although the paper acknowledges several limitations and qualifies the degradation as configuration-dependent, those qualifications do not establish why the configurations fail. I am not asking for a universal claim across all agents; even a single-system study could be informative if it isolated the mechanism convincingly. Here, the evidence supports a narrower finding about these implementations, while the broader scientific interpretation as agentic overthinking remains insufficiently substantiated.
  2. The dataset is selected and labeled in the same LLM pass, with null examples collected under a different instruction and a per-screenplay quota. Human agreement on the fine-grained labels is low: Appendix B reports human–human Cohen's kappa of approximately 0.181 for intent and 0.145 for theme. Moreover, retrieved context may support interpretations unavailable to the original local annotator. Evaluating every system against the same reference is useful for comparison, but does not ensure that reference bias affects all systems equally. Without an independent assessment of whether generated claims are supported by the evidence available to each system, reduced classification agreement cannot establish increased over-interpretation. This limits both the interpretation of the degradation and the proposed account of its causes.

Questions for authors
  1. What evidence distinguishes the reported degradation from deficiencies of the particular agent implementation? In particular, after controlling the prompt, decoding settings, controller adaptation, and feedback functionality, does additional deliberation still cause degradation? Please reconcile the claimed fixed-prompt 27.3%→13.0% comparison with Tables 1 and 6, which indicate different NFP settings. This is my main question because it determines whether the paper establishes more than an unsuccessful system configuration.

  2. Of 487 VERIFY actions, only 70 yield a parsed direction, and none yields SIMPLIFY. What happens in the remaining cases? Is this due to parsing, missing inputs, or another execution condition? How do the reported 95.2% held-out direction accuracy and the online behavior relate when evaluated on actual agent-generated candidates? Please distinguish the absence of an appropriate critic signal from failure to extract or apply it.

  3. Is there an independently assessed subset showing that answers changed by the loop contain unsupported claims, rather than alternative interpretations or interpretations supported by retrieved context? In particular, does the system ordering persist against independent judgments of claim support? Such evidence would materially affect my assessment of the proposed explanation.

  4. Given that the Llama controller is unadapted and NarrAgent fails to terminate on 60.8% of instances, what evidence supports interpreting its low scores as replication of overthinking rather than inadequate controller training? Does the degradation remain among valid, completed trajectories? Please distinguish evidence currently available from additional experiments proposed for future work.

Reviewer_vNagOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies agent overthinking on a cinematic subtext-detection task that requires pragmatic inference about what characters mean but do not state explicitly. The authors introduce three mechanisms—Confidence-Gated Retrieval, a Bidirectional Critic, and an Interpretation Depth Controller—and conduct experiments on the LitSubText dataset. However, these three mechanisms fail to address the agentic overthinking problem. Further experiments that fix the input and vary the length of the candidates show that the model can separate wrong candidates from correct ones, but not over-detailed ones from concise ones.

Critical strengths

Interesting experiments and findings on the studied task.

Critical weaknesses
  1. I cannot tell what is the novel conclusion of this work after go through the paper.
  2. The loop-specific attribution under a fixed prompt in Section 5.1 is inconsistent as reported in Section 5.2. This is the concern most likely to change my recommendation.
Questions for authors

What is the novelty and conclusion of this work? What is the overall optimal settings?

Reviewer_kRH4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LitSubText, a benchmark for classifying intent and thematic subtext in screenplay dialogue, and studies whether increasing agentic inference improves performance. Across the tested configurations, prompt-only inference outperforms the NarrAgent loops, while three proposed restraint mechanisms, which are gated retrieval, a bidirectional critic and a depth controller, all fail to recover the loss. Instrumented trajectories and controlled probes suggest that the critic primarily detects incorrect labels rather than excessive interpretation, and that its restraint signal rarely operates in deployment.

Critical strengths
  1. The experimental design is thoughtful. The authors formulate three hypotheses for why the agentic loop fails and investigate them through component ablations, instrumented trajectories, controlled critic and controller probes, and cost analysis.
Critical weaknesses
  1. The validity of LitSubText is insufficient to support the paper’s central conclusions. Each instance is simultaneously selected and labeled by a single model, while human agreement with the released intent/theme labels is only κ = 0.177/0.155, and human–human agreement is similarly low at κ = 0.181/0.145. This suggests that the 8-way/9-way taxonomy may not be reliably applicable even by humans. The claim that a fixed noisy label set cannot generate the observed system ordering is too strong, since systematic annotation and selection biases may affect different systems differently.
  2. The central inverse-scaling claim is substantially overstated. The experiments study only one custom agent framework rather than standard or external agent implementations, and only one backbone has a properly trained controller. More importantly, the agent baseline itself appears degenerate: the authors acknowledge that SFT taught the controller “to run the loop rather than answer the question,” NarrAgent predicts CC for 123 of 240 instances although only four are labeled CC, and every agent configuration falls far below the 26.3% majority baseline in theme accuracy. These results may therefore diagnose a weak custom agent implementation rather than establish a general inverse-scaling property of agentic inference.
Questions for authors
  1. Can the authors evaluate modern models and frameworks, including reasoning models such as Qwen3/Qwen3.5 or DeepSeek-R1-distilled models, to show that the phenomenon is not specific to a weak 7B controller and a custom agent loop?
Reviewer_bawxOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces LitSubText, a benchmark of dialogue excerpts from screenplays, to test whether language models can identify a character’s implied intent and a scene’s theme.

Critical strengths
  • It is interesting to see what the critic actually recommended when it reviewed the agent’s answers. Across 240 test passages, none of its 70 usable recommendations suggested a simpler reading. This helps explain why the safeguard did little to keep the agent from reading too much into the dialogue.
Critical weaknesses
  • Writing is confusing and hard to follow

  • Found Table 1 confusing as “L0 + narrative context” in the table and “L3” in the text refer to the same setup. Also, in Table 1, theme accuracy drops from 34.2% for L3 to 11.7% for NarrAgent. As NarrAgent uses a reasoning loop and a different prompt, it’s unclear whether the drop comes from the loop, the prompt, or both.

  • Found the analysis of the critic useful, but the main contribution was harder to follow. The title suggests a broader finding about when extra reasoning hurts performance, while the experiments mainly explain why one particular agent struggled to interpret film dialogue.

  • LitSubText accepts only one theme per scene, but human readers chose that theme in just 29.4% of their judgments (Table 4). This makes it hard to tell whether NarrAgent’s lower score means it misunderstood the dialogue or chose a different theme that also fits the scene.

Questions for authors
  1. In Table 1, theme accuracy drops from 34.2% for L3 to 11.7% for NarrAgent. Can you clarify whether that drop comes from the change in prompt, the reasoning loop, or both?

  2. In Section 5.2, why did only 70 of RA-NarrAgent’s 487 VERIFY actions result in a recorded recommendation from the critic? Could you clarify how the agent handled the remaining 417 checks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.