acceptodds
Withdrawn from ICLR 2027

When Does Acting Improve Learning? Humans, Language Models, and the Role of Thinking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Does learning from the same realized interaction depend on whether the learner chose the actions that produced it? We study this question in humans and language models using INTERGROUND, a matched-role paradigm in which an acting learner chooses consequential actions while an observer receives the same realized action–response record. Among 120 adults, acting improved transfer to novel and counterfactual situations by 5.8 percentage points (95% CI [2.6, 8.9]). Across 11 language model configurations spanning SmolLM2, Qwen3, Llama 3, and Gemma 3, however, standard online supervised adaptation produced only small acting–observer differences. In Qwen3, enabling thinking during learning increased the acting–observer gap across scales, with clear evidence at three of four tested scales; at 8B, the gap rose from 0.3 to 4.5 percentage points, a 4.2-point increase (95% CI [2.6, 6.4]). Randomizing whether the executed action matched the learner’s preferred action showed that thinking increased sensitivity to action–experience alignment. The same thinking-dependent pattern generalized to a structurally distinct causal-intervention environment at Qwen3-8B, where the gap increased from 0.4 to 2.9 points. Our findings show that matched trajectories do not guarantee matched learning: replaying an interaction alone does not always reproduce the benefit of generating it. For agent training, these results argue for retaining learner-specific context preceding each action, including reasoning and action preferences, alongside the observable trajectory.

Primary area:
applications to neuroscience & cognitive science
Keywords:
active learning, observational learning, language models, reasoning, causal learning, structural transfer, online adaptation, agent learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_L39yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces INTERGROUND, a matched actor–observer paradigm for studying whether learning depends on who generated an interaction, even when the realized trajectory is identical. In human experiments, acting improves structural transfer, whereas standard online supervised adaptation produces little actor–observer difference in language models. The authors further show that enabling pre-action thinking in Qwen3 increases the acting advantage and makes learning more sensitive to whether the executed action matches the learner’s own prior preference, suggesting that learner-specific reasoning can shape how subsequent experience is incorporated.

Critical strengths
  1. The research question is unusual.
  2. The paper is easy to understand.
  3. It has real-world test.
Critical weaknesses
  1. The near-zero actor–observer gap under thinking-OFF is partly built into the learning setup. Both learners receive the same realized trajectory, the same supervised targets, and the same number of updates, while the fact that an action was self-chosen is not explicitly represented in the supervised objective. Therefore, the conclusion that “action selection alone is insufficient” may be stronger than what this baseline establishes.

  2. Under thinking-ON, the actor and observer no longer receive the same effective learning input, making the effect difficult to attribute specifically to acting. Each learner generates its own private reasoning, which is retained as conditioning context for subsequent learning, so the observed gap may reflect reasoning–trajectory consistency, self-consistency, or on-policy conditioning rather than causal control itself. A stronger control would hold the reasoning prefix fixed across roles and vary only whether the learner’s action preference determines execution.

  3. The human and LLM observer conditions are not operationalized in the same way. Human observers passively replay another participant’s actions without forming their own action choices, whereas LLM observers still form an independent preferred action, and under thinking-ON also generate their own private reasoning. Thus, the human comparison is closer to active choice versus passive observation, while the LLM comparison is choice-with-control versus choice-without-control.

  4. The practical significance of the matched actor–observer setting is not sufficiently established. The paper extrapolates the findings to agent training and trajectory replay, but does not demonstrate that preserving pre-action reasoning or preferences improves learning in a realistic replay, offline-learning, or long-horizon agent setting. Since both evaluated environments are highly controlled, the broader implication for practical agent training remains speculative.

  5. The model coverage is limited and does not include frontier reasoning systems. Although the standard baseline includes several open-weight model families up to 14B parameters, the central finding that thinking increases the actor–observer gap is tested only within the Qwen3 family, and the second-environment replication is conducted only with Qwen3-8B. It is therefore unclear whether the effect generalizes to stronger frontier reasoning or agentic models.

  6. The human–LLM comparison is confounded by substantially different learning protocols. Humans receive only four five-turn episodes per hidden rule, whereas models receive many more episodes together with explicit online supervised LoRA updates. As a result, the difference between the human acting advantage and the near-zero LLM baseline may reflect differences in learning algorithm, exposure budget, or interface rather than a fundamental difference between human and model learning.

  7. The human study uses a between-subject design and does not fully control for individual differences in learning ability. Sixty participants are assigned to the acting role and sixty to the yoked-observer role, so trajectory matching controls the experienced interaction but not participant-level differences such as rule induction, working memory, or counterfactual reasoning ability. A counterbalanced within-subject design, or at least clearer evidence of randomized assignment and baseline comparability, would strengthen the causal interpretation of the human acting advantage.

Questions for authors
  1. How can causal authorship affect learning in the thinking-OFF setup if actor and observer receive the same trajectories, targets, and updates?

  2. Can the authors disentangle acting from learner-specific reasoning context, for example by holding the reasoning prefix fixed across roles?

  3. Why are the human and LLM observer conditions defined differently, and does this affect the human–model comparison?

  4. What realistic agent-learning scenario motivates this matched actor–observer setup, and does retaining reasoning or action preferences help in practical replay settings?

  5. Does the thinking-dependent acting advantage generalize beyond Qwen3 and to frontier reasoning models?

  6. How much of the human–LLM difference is due to different learning protocols and exposure budgets?

  7. Were human participants randomly assigned to roles, and how were individual differences controlled?

Reviewer_ARhwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework, INTERGROUND, that gives acting and observing learners the same experience: an acting learner chooses the actions in an interaction governed by a hidden rule, while an observer receives the same realized action-response record. Humans and models learn the same six hidden rules and solve the same two tasks, label identification and structural transfer. The authors find an acting-observer gap in the human experiment with 120 adults. In the model experiments, little gap appears under standard training. The authors then study the effect of reasoning (thinking) on the gap: the gap appears in models only when Qwen3's thinking mode is enabled during learning. They support this with an action-experience alignment manipulation, several control experiments, and a replication in a causal-intervention environment.

Critical strengths
  • The paper starts from an established finding in human active vs. yoked learning and asks whether it carries over to language models. This kind of human-experiment-motivated analysis is valuable and worth encouraging.
  • INTERGROUND gives acting and observing learners the same realized interaction record within each pair, and humans and models learn the same six hidden rules and are evaluated on the same two tasks, label identification and structural transfer. This makes the human and model studies directly comparable in their logic.
Critical weaknesses

[W1] The paper does not reach a decisive implication; after the initial result, the model sections are largely ablations of thinking ON/OFF

  • The paper's flow starts from a clear human result: acting improves structural transfer.
  • On the model side, however, there is no comparable gap under standard adaptation. The model-side effect therefore appears only when Qwen3's thinking mode is enabled during learning (Section 5).
  • The remaining sections then present a sequence of manipulations and controls, each of which accounts for only "part" of the effect, without converging on an explanation of what the gap means
  • As a result, the reader learns that a thinking-dependent gap exists in Qwen3, but not what it implies for how models learn from self-generated vs. replayed experience, or how it relates to the human acting advantage that motivates the study.

[W2] Running a thinking model with thinking OFF is unnatural, yet the ON-vs-OFF gap is framed as the central finding

  • Qwen3 is a thinking model, and running it with thinking OFF is not its natural operating mode. Yet the contrast between thinking ON and OFF is framed as the paper's central model-side finding.
  • Under thinking OFF, the learner does no pre-action computation at all: it directly scores the two candidate action.
  • Learners trained with thinking ON appear to be still evaluated with thinking OFF.
  • The "critical" contrast therefore compares the model's natural mode against a stripped-down one

[W3] The thinking analysis uses a single model family; the model pool should be narrowed toward models that better fit the human comparison

  • All thinking-related results come from a single thinking model family, Qwen3.
    • The causal-intervention replication uses only Qwen3-8B
    • The frozen in-context control only Qwen3-1.7B and 8B.
    • The paper qualifies its claims as holding "in Qwen3", which could be not held in other reasoning models.
  • Meanwhile, the 11-configuration in Table 1 adds breadth only to the standard condition, where no effect is found.
    • Recent flagship models would be a better fit for comparison with the human experiments.
    • I would recommend narrowing the pool of model experiments and focusing on what the gap means and what the analysis suggests, supports, and motivates.
    • The frozen in-context variant (Section 5.4), which requires no weight updates, could make testing larger models feasible.

[W4] The data used for training and evaluation, and the minimum details of the model experiments, are not presented in the main text

  • INTERGROUND is an in-house, procedurally generated environment, so the source of both the training data (interaction records generated with the rule engine) and the evaluation data (held-out configurations) is the paper's own construction.
  • The main text describes it only at a high level and defers its specification to Appendices.
    • There is no summary of how much of the small configuration space the teaching records cover.
    • LoRA is mentioned only in passing in the main text. What the models are trained on, how many updates each learner receives, the LoRA configuration, and how the "acting" model selects actions are all appendix-only. This hinders the validation of model results and understanding of this paper.
    • Furthermore, the learning rate is not reported anywhere.

[Minor] Too many confidence intervals; variance over runs should be reported directly

  • The main text reports a confidence interval (CI) for nearly every number, which makes the results hard to read.
  • Given the small evaluation sets, reporting the full variance over runs (e.g., mean and standard deviation) would be more transparent than percentile intervals.
Questions for authors

See weaknesses

Reviewer_BVJ7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work analyzes whether learning from an action-response record depends if the learner is choosing the action or acts just as an observer. In INTERGROUND an acting learner chooses actions while a yoked observer receives the same realized action-response. Two main kinds of task are tested: label identification (recovering the learned label of the rule behind a new trace) and structural transfer (applying the learned rule to novel partners and counterfactual histories). The authors report that in a study of 120 adults acting improves structural transfer by 5.8 percentage points, with similar performance in label identification. Experiments on 11 language model configurations under supervised adaptation show small gaps in structural transfer. When learning-stage thinking is enabled in Qwen3 these gaps widen, ranging from 0.9 percentage points (0.6B model) to 4.5 (8B model). Additional experiments are reported regarding randomly matching the executed action with the learner's preferred action, several controls (frozen in-context learning, test-only thinking, reasoning-conditioned updates, changing trajectory source) and a replication in a causal-intervention environment.

Critical strengths
  • The paper addresses an interesting question: what role does choosing actions play in learning, compared with just observing the same experience? Analyzing this in humans and language models within a common framework is a useful contribution to understanding learning from interaction.

  • The fact that reasoning has an impact on the acting-observer gap is relevant for agent training. It suggests that experience has different value depending on the learner's reasoning context, which is a useful observation with potential implications for how agents learn from their own interactions and replayed trajectories.

Critical weaknesses
  • It is not clear if the main test is really measuring learning that generalizes to new situations. In the paper it is stated that the tested situations are not part of the learning (for example from Section A.3, or D.5), but for example looking at Rule 1 in Table 7, both possible first moves (A and B) are test examples. Therefore, at least one example is overlapping with learning, since during learning every episode for the rule has to start with either A or B. The table reports a representative test inventory, but the paper would benefit from an explanation of how overlap is excluded in practice. This matters for interpreting the reported transfer results.

  • I find the reason for the model advantage still unclear. When thinking is off, actor and observer start from the same checkpoint and adapter initialization and are trained on the same training data. Up to randomness in training, it is expected that their updates are the same. Therefore, a small gap between roles is largely expected. Humans start with potentially different beliefs, making the comparison between human and LM results less informative.

    When thinking is on, each model's reasoning is included in its training. Let's consider the scenario where the actor reasons that A is its preferred choice, while the observer reasons that B is its preferred choice. A is then executed. The actor learns from its reasoning about A, followed by the results of A; the observer learns from its reasoning about B, followed by the same results of A. The actor may benefit because the feedback (results of A) concerns directly the hypothesis it was reasoning about (to prefer executing A); the observer may have had good reasons to prefer B, but receives feedback not related to it (still potentially informative). Allowing an observer to reason about A after knowing it was chosen by the actor, but before seeing the response, might improve its learning. This would help to understand whether suitable reasoning can help recovering the actor-observer gap.

Overall I find the exposition to be unnecessarily difficult to read and too abstract even when explaining straightforward concepts. Some terminology is used without introduction, paragraphs seem overly compressed, with awkward sentence constructions and unnecessary jargon without concrete descriptions. This makes the paper harder to follow than it needs to be.

Questions for authors
  • How are situations seen during learning excluded from the test? Is the actor's advantage still present when only new situations are considered in the test?

  • With thinking on, when is reasoning done, and what goes into the training inputs? A training example for both actor and observer would help.

  • If the actor uses reasoning to choose an action, but reasoning is then removed from the training inputs of both models, does the actor still perform better? (please report the results)

  • (Additional analysis) Does the observer do better if its reasoning traces are replaced with reasoning about the action actually chosen by the actor but before knowing the response? (using same reasoning length limits)

  • When thinking is off, are training inputs identical across roles? What contributes to the remaining difference in performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.