acceptodds
Under review as a conference paper at ICLR 2027

From Imitation to Understanding: Emergent Expert Modelling in Social Learning

Abstract

Recent work has demonstrated that artificial agents can develop emergent social learning, using observations of expert agents to solve otherwise difficult tasks without an explicit imitation objective. In humans, such imitation is strongly linked to Theory of Mind: the ability to model the intentions, mental states, and behaviours of others. Motivated by an information-theoretic analysis of the information offered by the expert, we study whether emergent social learning analogously induces expert-modelling capabilities: the representation of the expert's latent state, goal, and behaviours. We study this question across four qualitatively different environments covering grid-worlds (GoalCycle, Overcooked), continuous control (Brax Ant), and an abstract combinatorial task (Travelling Salesman). We then test this relationship empirically and report three findings. First, social learning tracks how informative the expert is: manipulations that make the expert redundant (e.g. revealing the goal to the learner) collapse social learning and expert modelling together. Secondly, without manipulations and across independently trained agents, stronger social learning is correlated with stronger expert modelling in almost every setting. Third, the modelling view suggests concrete interventions in the form of representation-learning objectives and curiosity-style rewards that lead to large increases in both social learning and expert modelling. Together, these findings suggest a strong developmental link between social learning and modelling others.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Social Learning, Theory of Mind, Expert Modelling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZSU7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines whether emergent social learning enables expert-modelling capabilities without explicit imitation objectives. It presents an information-theoretic framework and blind-corrected probing metrics to evaluate expert representations. Experiments in four environments (GoalCycle, Overcooked, Brax Ant, and Travelling Salesman) show a positive relationship between social learning and expert modelling, with targeted interventions enhancing both in several settings.

Critical strengths

The paper presents a well-motivated information-theoretic framework and blind-corrected probing metrics to distinguish expert-specific representations from shared environmental information. This systematic, interpretable approach advances the field by offering a clear causal account of emergent expertise, going beyond merely tracking improvements in social learning performance. The evaluation covers four distinct environments and uses controlled manipulations, correlation analyses, and targeted interventions to examine the relationship between social learning and expert modelling. The experimental breadth and transparent reporting of inconsistent findings enhance the credibility of the empirical contribution and support acceptance.

Critical weaknesses

The abstract could benefit from refinement to more clearly foreground the study’s novelty, particularly its theoretical and methodological contributions, which the main text articulates well, but the abstract's emphasis on empirical findings somewhat overshadows. Since the learner and expert generally share the same latent task, could the authors clarify how the observed expert-modelling capabilities differ from representations of task-relevant information inferred through expert observation, beyond what the blind-corrected probes and misaligned GoalCycle experiment already establish? Since earlier work, especially Bhoopchand et al. (2023) “Learning Few-Shot Imitation as Cultural Transmission”), has already examined internal representations of social information and goal states, could the authors more clearly explain how their study of emergent expert modelling goes beyond these analyses? The Halfway condition is evaluated using total episode returns; could the authors clarify how this metric distinguishes genuine in-context learning and retention from performance gains accumulated while the expert is still visible, and whether performance after the expert disappears was assessed separately? In Section 4, the information-theoretic analysis establishes a lower bound on expert dependency; however, could the authors clarify the assumptions under which improvements in social learning necessarily imply nonzero expert dependency, beyond the data-processing inequality established in Appendix F.1? In Section 5, social learning is measured across training checkpoints, while expert modelling is evaluated after training; could the authors clarify whether the reported correlations hold when both metrics are evaluated at corresponding training stages? The default Overcooked masking in Section 6 removes only the expert's symbol while retaining observations of its kitchen; could the authors clarify whether indirect information about the expert remains accessible in the Solo and blind conditions, and how this affects the reported expert-modelling results? In Section 7.3, the authors propose that interventions may improve exploration and representation learning more generally, and that alternative intrinsic rewards produce comparable benefits in Appendix G.6; could the authors clarify what evidence specifically supports expert modelling as the mechanism driving improvements in social learning? In Section 8, Brax shows no significant correlation between social learning and expert modelling, and that some interventions produce inconsistent results across environments; could the authors clarify how these findings support the broader conclusion that expert modelling and social learning are tightly coupled, and under what conditions this relationship is expected to hold?

Questions for authors

The abstract could benefit from refinement to more clearly foreground the study’s novelty, particularly its theoretical and methodological contributions, which the main text articulates well, but the abstract's emphasis on empirical findings somewhat overshadows. Since the learner and expert generally share the same latent task, could the authors clarify how the observed expert-modelling capabilities differ from representations of task-relevant information inferred through expert observation, beyond what the blind-corrected probes and misaligned GoalCycle experiment already establish? Since earlier work, especially Bhoopchand et al. (2023) “Learning Few-Shot Imitation as Cultural Transmission”), has already examined internal representations of social information and goal states, could the authors more clearly explain how their study of emergent expert modelling goes beyond these analyses? The Halfway condition is evaluated using total episode returns; could the authors clarify how this metric distinguishes genuine in-context learning and retention from performance gains accumulated while the expert is still visible, and whether performance after the expert disappears was assessed separately? In Section 4, the information-theoretic analysis establishes a lower bound on expert dependency; however, could the authors clarify the assumptions under which improvements in social learning necessarily imply nonzero expert dependency, beyond the data-processing inequality established in Appendix F.1? In Section 5, social learning is measured across training checkpoints, while expert modelling is evaluated after training; could the authors clarify whether the reported correlations hold when both metrics are evaluated at corresponding training stages? The default Overcooked masking in Section 6 removes only the expert's symbol while retaining observations of its kitchen; could the authors clarify whether indirect information about the expert remains accessible in the Solo and blind conditions, and how this affects the reported expert-modelling results? In Section 7.3, the authors propose that interventions may improve exploration and representation learning more generally, and that alternative intrinsic rewards produce comparable benefits in Appendix G.6; could the authors clarify what evidence specifically supports expert modelling as the mechanism driving improvements in social learning? In Section 8, Brax shows no significant correlation between social learning and expert modelling, and that some interventions produce inconsistent results across environments; could the authors clarify how these findings support the broader conclusion that expert modelling and social learning are tightly coupled, and under what conditions this relationship is expected to hold?

1 comment

We thank the reviewer for the thoughtful review. We respond to the reviewer's concerns below. We've marked the changes in blue in the revised version of our manuscript.


ZSU7-W1 Abstract

We have revised the abstract to better emphasise the theoretical and methodological contributions. It now leads with the result that social learning requires encoding expert observations, then introduces the blind-corrected probes and the three hypotheses they test.


ZSU7-Q1 Expert vs task modelling

Task information alone would not let the learner predict the expert's momentary behaviour, such as its current action, which is what we primarily measure. For example, inferring the recipe in Overcooked requires modelling the expert's actions and positions, which is not reducible to the recipe itself. In the same way, ΔSubtask is positive even though only one subtask (choosing ingredients) reveals the recipe. Finally, when the task is given to both the learner and the blind baseline (H1), task information cancels out of ΔX-Pred, yet modelling stays above zero.


ZSU7-Q2 Relation to Bhoopchand et al.

We now discuss this in Related Work. Their work is complementary to ours. They probe whether the agent encodes the expert's presence and the agent's own goal entries. As they note, an auxiliary attention loss is at least partly responsible for the expert-presence representation.

By contrast, we probe unobserved expert properties (e.g., actions, hidden states, and subtasks), without an expert-directed objective, correct against an expert-masked baseline, and relate these representations to social learning across conditions and environments.


ZSU7-Q3 Retention

We initially adapted the protocol and metric from Bhoopchand et al. (2023). As you say, only the gain after the midpoint isolates retention. We now clarify this in Section 3.3. We also measure retention in App. E.1. The share of the second-half gain kept once the expert leaves is high in TSP, partial in GoalCycle 7×7 and near zero in Brax, and in Overcooked and GoalCycle 13×13 the seeds with the highest social learning retain much less than moderate ones.


ZSU7-Q4 Assumptions

We have added a paragraph to App. F.1 stating the assumptions explicitly: (a) the expert can influence the learner only through the expert-observation channel; (b) the learner's policy and critic are functions of its latent alone; and (c) the latent is updated recurrently. The learner's returns can then differ from Solo only if its actions depend on expert observations, which must pass through the latent encoding.


ZSU7-Q5 Matched training stages

Yes. With the same controls as Fig. 4 and both metrics averaged over training, every Fig. 4 correlation stays positive at similar strength, with all but one BCa interval excluding zero; Brax stays null. With both taken from the final evaluation, the GoalCycle correlations are unchanged and Overcooked is weaker. TSP's evaluation SLM is too noisy to rank seeds reliably.


ZSU7-Q6 Overcooked masking

We have expanded the Overcooked description in App. C. Solo and the blind baseline mask the expert's position, direction and inventory, so its kitchen remains visible (e.g. ingredients it has already placed in its pot). The recipe indicator is masked in both kitchens in every condition.

This makes both measures conservative. The blind baseline sees the same kitchen, so ΔX-Pred counts only information beyond it. Any cue the kitchen gives the Solo learner raises the Solo baseline, lowering SLM.


ZSU7-Q7 Mechanism

Prediction-based interventions could act through better exploration or representation learning in general, and we do not fully separate these from the effect on expert-modelling. Several results nevertheless point to expert-modelling being a key factor:

  • Legibility adds no task information and leaves the learner's objective unchanged, only making the expert easier to interpret, yet it raises SLM and ΔEmbedding (Fig. 6, shaded panel).
  • Social learning rises only where modelling does: where an objective fails to raise modelling (Overcooked), SLM does not rise either.
  • Undirected exploration, an entropy bonus, does not help (App. G.6).
  • In Overcooked, the expert is always in view, yet surprisal still strongly increases SLM and the modelling metrics.

ZSU7-Q8 Brax and conditions

Intervention effects varied across environments. Across 13 of 14 comparisons in Figure 6, social learning and expert modelling either both improved or neither did. In Brax, both SLM and expert modelling declined when the expert became less informative (H1) and improved under an intervention (H3). The across-seed correlation in Brax remains inconclusive (H2) because Expert Modelling varied little across seeds, possibly because learners could directly observe the expert’s state. We expect the two measures to move together when the expert’s behaviour provides information the learner needs to solve the task.

Reviewer_vr1JOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether agents trained for their own rewards acquire representations of an expert through observational social learning, without an explicit imitation objective. It combines an information-theoretic account of expert dependency with linear probes corrected by an expert-masked baseline. Experiments span GoalCycle, Overcooked, TSP, and Brax Ant. The authors report that reducing the expert's informativeness weakens both social learning and modelling; that the two correlate across independently trained learners in most settings, with Brax an explicit exception; and that representation objectives, curiosity rewards, or a more interpretable expert view can improve both. Goal and subtask probes extend the analysis beyond immediate actions. The main contribution is a systematic empirical account of the connection between benefiting from an expert and representing its behaviour and goals.

Critical strengths

The three hypotheses give the experiments a clear structure, and the appendices provide useful implementation and evaluation details.

  1. The paper asks whether agents pursuing their own rewards spontaneously develop representations of an observed expert, without an explicit imitation objective. The one-way, non-interactive learner–expert setting makes this question precise and experimentally tractable. I particularly appreciate the simplicity of the setup! It allows the paper to investigate a conceptually interesting question without introducing unnecessary architectural complexity.

  2. The paper develops its central question through complementary tests of expert informativeness, relationships across independently trained learners, and interventions on representations and intrinsic rewards. The expert-masked baselines and controls for returns and co-presence make the probe results more informative. Testing across four different environments, while also reporting exceptions and mixed results, provides a nuanced account of when social learning is accompanied by expert modelling.

Critical weaknesses
  1. Terminology could be made more consistent. Section 5.3 describes ΔSubtask-Pred as predicting the expert’s “next sub-task,” whereas Figure 5 and Appendix E.4 refer to its “current sub-goal.” These descriptions may refer to the same intended operation, but consistent wording would make the prediction target clearer.

  2. Minor typographical errors. “Varianbles” on p. 4 should be “variables,” and “seperate” in Appendix E.3 should be “separate.”

Questions for authors

Could you clarify the temporal meaning of the ΔSubtask-Pred target and harmonize its description across Section 5.3, Figure 5, and Appendix E.4? In particular, does “next sub-task” refer to the expert’s currently intended operation, as identified by the state-derived label, rather than a future subtask transition? This is a terminology clarification; I am not requesting additional experiments.

2 comments

We thank the reviewer for a thoughtful and constructive review. We have addressed the concerns and revised the draft (changes are marked in blue).


vr1J-Question + W1

Yes, "next sub-task" refers to the expert's currently intended operation, i.e. the step it must complete to proceed with cooking. For example, if the expert has just placed the final ingredient in the pot, the "next sub-task" is now to pick up the plate. As you suggested, we have harmonised the description of this across the revised paper (to always refer to it as the current subtask).


vr1J-W2

Thanks for pointing these out, we have fixed these in the revised version


Reviewer_vr1JOpenReview ↗

Reply

Thanks for the reply! Your changes successfully addressed my concerns. I will keep my rating the same since 4 is the highest rating possible ;)

Good work~

Reviewer_PB3HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Reasoning about other agents is important, not just for effective collaboration, but efficient learning (e.g., from experts). The authors here study whether and how modeling of experts emerges alongside general social learning capacities. The authors explore these questions through a mix of theoretical and empirical work in a reasonably diverse set of environments. The authors generally find that expert modeling and social learning are correlated and involve (in part) tracking how informative an expert is, and that increased modeling of experts can increase social learning.

Critical strengths

I think the authors make a nice contribution in highlighting relations between social learning and expert modeling. I think this question (especially around expert modeling) will only grow as AI systems are potentially deployed in multi-agent contexts and the authors' theoretical work can help understand richer notions and relations between different kinds of learning / other modeling.

I appreciate that the authors approach this through a mix of theoretical and empirical work. I think the insiht that social learning and expert modeling can collapse together is particularly interesting. The authors also do a good job highlighting their limitations transparently.

Critical weaknesses

While excitingly the authors do so much in the paper, at times I felt that the main text was quite clunky (especially around the theory leading into the experiments). I had some trouble following the experimental results. It might be worth doing a bit more signposting for readers throughout of what the key contributions and insights are (indicated in my slightly lower confidence rating). This, I think, is a generally nice problem for the authors to have though.

I did have some questions on some of the results: I was surprised that the CIs are so wide for some of the experiments (e.g., Fig 4 compared to Fig 6?) The authors note that the 95% CIs are over the 25 seeds per condition -- is this averaging in any way before the 95% CIs? How many samples are drawn (is it non-parameteric sampling?) I am open to revising my score here, but feel a little wary of endorsing acceptance just yet.

It also may help to have one or more qualitative examples of some of the behavior in the enviornments the authors discuss? (for space, this could be in the appendix). But more concrete examples of behavior could help a reader really understand what is going on.

Questions for authors
2 comments

We thank the reviewer for a thoughtful and constructive review. We respond to the concerns below. We have marked changes in blue in the revised version. We welcome any further feedback.


PB3H-W1 Connecting the theoretical motivation to the experiments and clarifying the findings

Thank you for this feedback. We have added signposting throughout the paper (all are marked in blue):

  • Abstract: We state the three hypotheses and their findings.
  • Introduction: We link each hypothesis to its results subsection.
  • Section 3: We open it with an overview.
  • Section 4: We motivate H1–H3 from the theoretical view (lines 239–243).
  • Section 7: Each experiment opens with its hypothesis and main finding, and Section 7.3 notes where the interventions were less effective.
  • Discussion: We open with an overall-findings summary. We would continue working to improve clarity for the camera-ready version.

PB3H-W2 Confidence intervals in Figures 4 and 6

Figure 4 shows partial correlations across seeds, while Figure 6 shows mean metric values. With 25 seeds, moderate correlations can have wide confidence intervals. The two figures estimate different quantities, so their interval widths are not directly comparable. For each seed, we split the rollouts collected during training into 100 buckets, and compute SLM for each bucket. We then average SLM over these buckets. Intuitively, each bucket represents a snapshot in SLM development during training. We average the expert modelling metrics on 100-10,000 evaluation episodes using the final checkpoint, depending on the tasks.


PB3H-Q1 Bootstrap procedure and sensitivity

The original Figure 4 didn't use bootstrapping. It used analytical Fisher-z confidence intervals. To address your question about sensitivity to the set of seeds, we have revised Figure 4, and it reports 95% bias-corrected and accelerated (BCa) bootstrap intervals. We use 10,000 resamples of the seed-level data.

The means are unaffected. The intervals from the bootstrap are generally slightly wider. However, the conclusion of the results doesn’t change from the original figure. Every interval that excluded zero still does, and TSP ΔEmbedding and Brax (both metrics) still include zero.

The largest change is in GoalCycle 13×13. Its correlations remain high (partial r = 0.81 for ΔAction and 0.91 for ΔEmbedding), but their intervals widen considerably while still excluding zero (ΔAction [0.59, 0.92] → [0.27, 0.97]; ΔEmbedding [0.80, 0.96] → [0.11, 0.99]).

For GoalCycle 13×13, the Fisher-z intervals were too narrow here because the data are highly skewed. This is mainly because social learning is less likely to arise naturally in this setting, as the co-occurrence between the learner and the expert is sparser than in 7×7. A few out of 25 seeds drive the strong positive correlations.


PB3H-W3 Qualitative example of social learning

We added a sequence of snapshots from an Overcooked episode in Appendix G.7 (Figure 13). The learner first prepares an incorrect dish, then begins selecting the same ingredients as the expert. After the expert is hidden, the learner adds the final ingredient and serves the correct dish. We also report how often this process fails: the learner often only partially retains the recipe shown by the expert.


PB3H-Q2 Joint inference and social transmission

Thank you for suggesting highly relevant and very interesting works. We incorporated all the suggested references across Related Work and Limitations. In Related Work, we connect joint inference in pedagogical interactions to our study of how expert informativeness shapes social learning.

In Limitations, we now discuss how selectiveness (highlighted by those works) is a key feature of social learning, in part because it helps to resolve Rogers' paradox (Collins et al., 2026). We also added discussion of how our work focuses on the learner's side of the interaction, and that studying the teacher's side is an interesting direction for future work.


Reviewer_PB3HOpenReview ↗

Strong response

Dear Authors,

Thank you for your response! You have helped address several of my concerns. I think the revised paper is quite a bit stronger, and I appreciate the effort the authors put into doing so. I have raised my score accordingly.

Comments on the paper

1 comment

My concerns have been addressed. No further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.