acceptodds
Under review as a conference paper at ICLR 2027

RATFtouille: Evaluating Emotional and Goal-Directed Reasoning Capabilities of Language Models Using Cooking Recipes

Abstract

While much of the current study of LLM reasoning is focused on goal-directed reasoning and emotional reasoning as two separate tasks, there are many real-world scenarios where LLMs are required to do them jointly. For example, if a user deletes a photo and then expresses strong sadness, an assistant should infer that the deletion was likely unintended and that the user wants the photo recovered. In cases like these, an LLM must infer a user's emotional state and use that inference to perform reasoning on their goals. In order to systematically evaluate LLMs abilities to perform emotionally contextualized goal-directed reasoning (ECGD), we propose the Recipe-Actor Transcript Framework (RATF), a framework that procedurally generates step-by-step transcripts of simulated actors following recipes to make food, where the recipes are sourced from the RecipeNLG dataset. We utilize these transcripts to test whether an LLM can correctly identify the intents and goals of a simulated actor, and predict their next action. More precisely, we apply RATF to study two LLM abilities: (1) goal prediction, where LLMs must infer the actor's goal given their past behavior, i.e., what are they trying to cook, and (2) next-action prediction, where we evaluate the accuracy of the LLM's predicted next-action distribution of the simulated actor when given in-context examples of actor behavior. We test several frontier LLMs and find that LLMs struggle with these tasks, and their performance is highly sensitive to surface-level variations in the way the prompt is presented. We find evidence suggesting that LLMs rely on heuristics to solve this task rather than performing true ECGD reasoning.

Primary area:
datasets and benchmarks
Keywords:
procedural generation, emotional reasoning benchmark, goal-directed reasoning benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_Hu6mOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RATF, a framework that generates cooking transcript data to test the capacity of LLMs on two reasoning tasks: goal prediction (i.e., what the actor is trying to cook) and next-action prediction based on the emotion cues. The framework takes real recipes from the RecipeNLG dataset, and generates step-by-step transcripts of a simulated actor cooking them. When generating simulated transcripts, the actor is initialized with a personality including 3 components: the probability of making and noticing a mistake; the probability of fixing the mistake and continuing; and if noticing the mistake, whether to stay calm and restart or be angry and give up. Based on the generated dataset, the paper investigates the capacity of LLMs on the two reasoning tasks, shows their limitations in predicting the simulated actors’ goals based on the emotions and actions, and analyzes their errors.

Critical strengths
  • Reasoning about goals and the next action based on emotion is an interesting and crucial research topic
  • The paper provides new information about how well the frontier LLMs (e.g., GPT-6-Astra) do in the proposed tasks, and provides insights about the errors.
  • The evaluated models cover strong and recent LLMs, e.g., GPT-6-Astra and DeepSeek-V4-Pro
Critical weaknesses
  • The generated framework is not evaluated with human subjects. One concern I have is whether what the authors expect from the LLMs is the same as what humans will actually do in this task, given the same prompts given to the LLMs. For example, a person can be shocked for many reasons, so whether it is an indicator of putting the wrong ingredient is uncertain, and which ingredient is the wrong one might be another question. The paper’s own results (Figure 5) are consistent with this: accuracy is much higher when the reaction is replaced with “has made a mistake”, which may reflect ambiguity in the emotional cues rather than a limitation of the LLMs.
  • The motivation of the setup of the next-action prediction task is unclear. Since the actions come from a simulated actor, which may not reflect how humans actually act and feel, it is unclear what this setting tells us about LLMs' ability to reason about humans.
  • The paper does not define the metric that compares the predicted and empirical action distributions in the next-action prediction task.
  • The size of the generated data and its statistics (e.g., transcript lengths, number of steps) are not clearly reported
Questions for authors

Besides the points in the weaknesses, I have the following question. Since the paper focuses only on the cooking domain, the findings may not generalize to other domains. As the generation process is automatic, which other domains do the authors think this framework can be applied to, and would the findings still hold?

2 comments
AuthorsOpenReview ↗

Response from Authors (W1-4)

We would like to thank the reviewer for their time and feedback. We respond to each of their concerns below. We ask the reviewer to let us know if there are any questions or concerns, so that we have the opportunity to further improve the paper. Revisions made to the paper have been highlighted in blue in the updated draft.

W1: Human Evaluation Data

We are currently in the process of running a human evaluation study testing human performance on the 2 and 4 option settings of the mistaken ingredient test (both standard and ambiguous variations) as well as the 10 and 20 in-context example settings of the next action test. While humans are currently outperforming models, we are waiting on more data to give a conclusive results. We commit to include these results in our full report by the camera-ready deadline.

W2,3: Next Action Disambiguation

As discussed in Line 351-353,

For each experiment, we adjust the number of occurrences of each mistake reaction in the full prompt to ensure the empirical distribution of mistake reactions is identical to the true mistake reaction distribution of the actor.

We modified the experiments in Section 3.2.2 (Personality-Based Next Step Prediction Methods) so "the models are asked to rank a list of three possible reactions from most to least likely" (Lines 343-344). The model can only reasonably be expected to learn the empirical freqencies from context as we do not provide any other information. Thus, the model's ranking of mistake reactions is evaluated on how well it matches the empirical distribution from the examples given in-context. We apologize for any confusion caused by unclear writing; these concerns have since been addressed in the revised paper in Sections 3.2.2 and 4.2 (Personality-Based Next Step Prediction Results).

W4: Generated Data Statistics

We have updated the paper with the relevant statistics in Table 3 of Section B (Experiment Parameters) in the Appendix, as seen below:

ParameterMissing Ingredient TestNext Action Test
# of samples5020
# of ingredients per recipe8-168-16
Mistake prompt transcript length4 (10 in LEGO domain)4
In-context transcript lengthN/A<32

where "mistake prompt transcript refers to the final transcript in each prompt from which models are asked to either predict the actor’s goal in the mistaken-ingredient experiment or to rank the actor’s mistake reactions in the next-action experiment" (Table 3 Caption, Line 771).

We apologize for any confusion caused the lack of clarity.

AuthorsOpenReview ↗

Response from Authors (Q1-2)

Q1: Addition of Two Domains alongside Cooking

To address the limited scope of our existing results, we expanded our current tests to include data from two other domains, simulating actors following LEGO instructions and wet lab protocols, respectively.

We discuss the integration of additional domains in Lines 67-75 in Section 1 (Introduction):

In order to test LLMs on emotionally contextualized goal-directed (ECGD) reasoning, RATF generates transcripts of simulated actors performing stepwise tasks across three domains (cooking recipes, LEGO builds, and wet lab protocols) and occasionally performing steps that evoke emotional reactions. For ease of discussion, we use standardized vocabulary across all domains: We refer to a set of step-by-step instructions as a recipe and the constituent inputs (i.e., ingredients, reagents, LEGO blocks) as ingredients. Initial recipe data comes from ground truth datasets per domain: RecipeNLG for cooking (Bie´n et al., 2020), Bricks for the Blind transcripts for LEGO (bri), and WLP-Dataset for wet lab protocols (WLP; Kulkarni et al., 2018).

Dataset processing of these domains is discussed on Lines 206-211 of Section 3.1 (Framework Design):

Each domain dataset is structured slightly differently, but we decided to but we decided to follow the formatting of RecipeNLG dataset, which contains a set of recipes, each with a title, a list of ingredients, and a list of natural-language instructions. We used gpt-6-astra\texttt{gpt-6-astra} to extract the ingredients and step-by-step instructions from the LEGO and WLP domains and reformat the dataset to resemble RecipeNLG; we refer to these datasets as ingredient-instruction datasets. The system prompts for this process can be found in Figure 13 of the Appendix.

Results for mistaken-ingredient and next action comparisons can be found in Figures 6 and 7 respectively. Below is a table of accuracies when performing mistaken-ingredient experiment averaged across models and broken down by domain, as shown in Figure 6:

SettingCookingLEGOWet Lab Protocol
2-option unambiguous0.0740.0950.072
4-option unambiguous0.3500.3860.136
2-option ambiguous0.0050.0560.060
4-option ambiguous0.0040.0560.016

We found that our results were consistent across domains, demonstrating that the behavior shown in our cooking results applies in broader instruction-following contexts. We hope these results narrow the gap between our evidence and conclusions.

Reviewer_5D3mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RATF procedurally generates cooking transcripts of simulated actors to evaluate emotionally contextualized goal-directed reasoning (ECGD), via recipe identification and next-action prediction. The paper reports LLMs struggle on both tasks and use ingredient-matching heuristics rather than ECGD reasoning.

Critical strengths

S1. RATF generates new transcripts from actors with adjustable personality parameters, and the options are built to stay "the same size and at the same edit distance from I". "All code and data generation scripts are freely available", so the benchmark can be regenerated and extended.

S2. Every model does best with the non-emotional baseline "[actor] has made a mistake", so the failure lies in inferring the mistake from the emotional reaction. The error analysis agrees: all 10 sampled 2-option DeepSeek-V4-Pro errors are "Ignores reaction".

Critical weaknesses

W1. The paper states that "The only way the LLM can answer the question correctly is if it bases its reasoning off of the emotional reaction", but every transcript ends with "[NA] added some [i′]. [NA] [R]", where R is always a negative reaction signalling a mistake, so in the standard setting simply excluding the last-added ingredient also picks the correct recipe. A control condition with neutral or positive reactions, where i′ is not a mistake, would show whether the benchmark actually requires reading the emotion.

W2. No human baseline is reported for either experiment, so it is unclear how reliably people infer a mistake from the templated reactions, e.g., "freezes in disbelief" versus "has made a mistake". A small human study on a subset of Experiment 1 items would show how much of the gap is an LLM limitation and how much is ambiguity in the reactions themselves.

Questions for authors

See Critical Weaknesses

1 comment
AuthorsOpenReview ↗

Response from Authors (W1-2)

We would like to thank the reviewer for their time and feedback. We respond to each of their concerns below. We ask the reviewer to let us know if there are any questions or concerns, so that we have the opportunity to further improve the paper. Revisions made to the paper have been highlighted in blue in the updated draft.

W1: Emotional Reaction Variation

We ran an experiement, shown in Figure 6, which shows the models’ sensitivity to specific reactions in the unambiguous mistaken ingredient setting. There, we measured the emotional reactions against a baseline of a non-emotional mistake reaction ("[NA] has made a mistake"). We have included the results of the 2-option setting in the table below:

Reactiongpt-6-astragpt-5gpt-5-miniDeepSeek-V4-Pro-0813Qwen3.5-9B
frowns with concern0.1800.1000.1000.0800.040
shakes their head slowly0.6120.0610.0410.0820.041
is visibly frustrated0.0400.0800.0400.0400.020
appears shocked0.1800.0800.0200.0400.000
freezes in disbelief0.0600.0800.0800.0800.060
has made a mistake0.9600.6400.4800.2200.440

We discuss these results in Lines 431-447 of Section 4.1 (Recipe Identification with Mistaken Ingredient):

In both settings, all models perform best when the prompt explicitly states that the actor made a mistake. This indicates that LLMs do not always identify that the actor has made a mistake in adding the last ingredient based on their emotional reaction. Notably, models which perform poorly given emotional reactions are still able to achieve high accuracy with non-emotional reactions; this shows that models which excel at goal-directed reasoning are not able to extend that skill to ECGD reasoning scenarios.

We hope that this demonstrates that the model's failures in these tasks can be attributed to model's (in)ability to use emotional reactions to contextualize the actor's goals.

W2: Human Evaluation Data

We are currently in the process of running a human evaluation study testing human performance on the 2 and 4 option settings of the mistaken ingredient test (both unambiguous and ambiguous variations) as well as the 10 and 20 in-context example settings of the next action test. While humans are currently outperforming models, we are waiting on more data to give a conclusive results. We hope to include these results in our full report by the camera-ready deadline.

Reviewer_ru4nOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RATF, which generates simulated actors’ action transcripts from recipes and evaluates models on mistaken-ingredient identification and personality-driven next-action prediction. The authors find that performance is affected by the number of candidate options, ingredient overlap, and reaction wording, and that historical examples from the same actor do not consistently improve behavioral distribution prediction. Based on these findings, the authors suggest that models may rely on heuristics and remain limited in emotionally contextualized goal-directed reasoning.

Critical strengths
  1. The research question has practical relevance. Using emotions as evidence to infer intentions and future behavior is a model capability worth investigating.

  2. The procedural framework enables control over candidate answers, reaction expressions, and actor parameters, while reducing the risk that complete test transcripts appear in training data.

Critical weaknesses
  1. The cooking setting is only weakly integrated with the tasks’ underlying computational structure, and its narrative framing exceeds its methodological contribution.

  2. The experiments do not clearly attribute model failures to deficiencies in the target capability, leaving the central conclusion insufficiently supported. The candidate-option experiments simultaneously alter option similarity, ingredient frequency, and answer distinctiveness, while the next-action experiments conflate behavioral understanding, finite-sample estimation, and generation distribution calibration. Without paired interventions on emotional reactions and simple heuristic baselines, the results establish only that models perform inconsistently on specific synthetic tasks. They do not sufficiently support the broader conclusion that models’ general emotional and goal-directed reasoning capabilities are highly limited.

Questions for authors
  1. What makes the cooking setting necessary for this evaluation?

  2. Have the authors conducted controlled comparisons that keep the action transcript and candidate options fixed while changing only the emotional reaction? Do the results establish that models actually use emotional information?

2 comments
AuthorsOpenReview ↗

Response from Authors (W1)

We would like to thank the reviewer for their time and feedback. We respond to each of their concerns below. We ask the reviewer to let us know if there are any questions or concerns, so that we have the opportunity to further improve the paper. Revisions made to the paper have been highlighted in blue in the updated draft.

W1: Cooking Setting Integration

We agree that the cooking setting is not particularly well-integrated with our testing framework, nor is the cooking setting particularly integral to our results. In truth, we designed RATF to work with multiple types of stepwise tasks and chose cooking as our setting as we found the RecipeNLG dataset to be well-organized and easy to work with. Cooking was also natural setting in which humans follow step-by-step instructions and sometimes make mistakes which lead to outward emotional reactions. In lieu of more deeply integrating cooking into our framework, we expanded our current tests to include data from two other domains, simulating actors following LEGO instructions and wet lab protocols, respectively.

We discuss the integration of additional domains in Lines 67-75 in Section 1 (Introduction):

In order to test LLMs on emotionally contextualized goal-directed (ECGD) reasoning, RATF generates transcripts of simulated actors performing stepwise tasks across three domains (cooking recipes, LEGO builds, and wet lab protocols) and occasionally performing steps that evoke emotional reactions. For ease of discussion, we use standardized vocabulary across all domains: We refer to a set of step-by-step instructions as a recipe and the constituent inputs (i.e., ingredients, reagents, LEGO blocks) as ingredients. Initial recipe data comes from ground truth datasets per domain: RecipeNLG for cooking (Bie´n et al., 2020), Bricks for the Blind transcripts for LEGO (bri), and WLP-Dataset for wet lab protocols (WLP; Kulkarni et al., 2018).

Dataset processing of these domains is discussed on Lines 206-211 of Section 3.1 (Framework Design):

Each domain dataset is structured slightly differently, but we decided to but we decided to follow the formatting of RecipeNLG dataset, which contains a set of recipes, each with a title, a list of ingredients, and a list of natural-language instructions. We used gpt-6-astra\texttt{gpt-6-astra} to extract the ingredients and step-by-step instructions from the LEGO and WLP domains and reformat the dataset to resemble RecipeNLG; we refer to these datasets as ingredient-instruction datasets. The system prompts for this process can be found in Figure 13 of the Appendix.

Results for mistaken-ingredient and next action comparisons can be found in Figures 6 and 7 respectively. Below is a table of accuracies when performing mistaken-ingredient experiment averaged across models and broken down by domain, as shown in Figure 6:

SettingCookingLEGOWet Lab Protocol
2-option unambiguous0.0740.0950.072
4-option unambiguous0.3500.3860.136
2-option ambiguous0.0050.0560.060
4-option ambiguous0.0040.0560.016

We found that our results were consistent across domains, demonstrating that the behavior shown in our cooking results applies in broader instruction-following contexts. We hope these results narrow the gap between our evidence and conclusions.

AuthorsOpenReview ↗

Response from Authors (W2)

W2: Model Failure Attribution

We ran an experiement, shown in Figure 6, which shows the models’ sensitivity to specific reactions in the unambiguous mistaken ingredient setting. There, we measured the emotional reactions against a baseline of a non-emotional mistake reaction ("[NA] has made a mistake"). We have included the results of the 2-option setting in the table below:

Reactiongpt-6-astragpt-5gpt-5-miniDeepSeek-V4-Pro-0813Qwen3.5-9B
frowns with concern0.1800.1000.1000.0800.040
shakes their head slowly0.6120.0610.0410.0820.041
is visibly frustrated0.0400.0800.0400.0400.020
appears shocked0.1800.0800.0200.0400.000
freezes in disbelief0.0600.0800.0800.0800.060
has made a mistake0.9600.6400.4800.2200.440

We discuss these results in Lines 431-447 of Section 4.1 (Recipe Identification with Mistaken Ingredient):

In both settings, all models perform best when the prompt explicitly states that the actor made a mistake. This indicates that LLMs do not always identify that the actor has made a mistake in adding the last ingredient based on their emotional reaction. Notably, models which perform poorly given emotional reactions are still able to achieve high accuracy with non-emotional reactions; this shows that models which excel at goal-directed reasoning are not able to extend that skill to ECGD reasoning scenarios.

We hope that this demonstrates that the model's failures in these tasks can be attributed to model's (in)ability to use emotional reactions to contextualize the actor's goals.

Reviewer_mfpWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces the Recipe-Actor Transcript Framework to evaluate how language models use emotional and behavioral cues to infer actor's goals. It generates cooking transcripts from RecipeNLG and studies two tasks. One identifies an intended recipe after an apparent ingredient mistake, and another one predicts an actor's reaction distribution from previous transcripts. The experiments vary candidate ambiguity, option count, reaction wording, and the relationship between the example actors and the target actor.

Critical strengths
  • The problem is interesting and worth studying. A helpful assistant may need to infer what a person wants from their reaction, rather than wait for an explicit instruction. Treating emotion as evidence for a hidden goal is a useful evaluation target. The motivation is relevant beyond cooking, even though the current experiments do not yet establish that broader applicability.

  • The controlled task is well designed. RATF combines observed reactions, partially observed actions, and candidate recipes in a setting where ambiguity can be changed systematically. It also introduces actors with controlled reaction policies. Compared with the original Event2Mind task, using the reaction as an input for goal inference is a meaningful distinction. The contribution is a specific diagnostic setting and its behavioral findings, not the general idea of studying emotions and goals together.

Critical weaknesses
  • I found that all experiments use synthetic cooking scenarios and a simple actor policy. Testing several models does not establish coverage of different social situations. The results support a finding about this controlled setting, but not yet the broad conclusion that LLMs have highly limited emotionally contextualized goal-directed reasoning. My concern is the gap between the scope of the evidence and the scope of the conclusion.

  • The related work part draws too sharp a line between RATF and earlier work. DynToM models how beliefs and emotions affect intentions and actions, and includes questions about the causes of mental-state changes. EmoBench includes Emotional Application, where emotion understanding informs the choice of a response or action. I believe that earlier work does not generally stop at recognizing emotion. I see moderate novelty in this setting, but the paper should explain what additional failure it identifies beyond these existing tasks.

  • Some visual details also need revision. Labels in Figure 2 and the legends/model names in Figures 5–6 are small relative to the main text. Figure 6 refers to four conditions but displays three, and its distance scale is not defined precisely. These issues make the results harder to verify.

Questions for authors
  • Can a small independent checked subset establish that readers interpret the intended mistake and identify the valid answers? On the same subset, can you compare model performance with a symbolic exclusion baseline, and test whether a non-emotional task with matched information and candidate structure shows the same option-count pattern?

  • Is the target the empirical context histogram or the latent simulator policy? Does the conclusion about actor-history use hold when evaluated against the simulator policy rather than only the context frequencies?

2 comments
AuthorsOpenReview ↗

Response from Authors (W1-2)

We would like to thank the reviewer for their time and feedback. We respond to each of their concerns below. We ask the reviewer to let us know if there are any questions or concerns, so that we have the opportunity to further improve the paper. Revisions made to the paper have been highlighted in blue in the updated draft.

W1: Addition of Two Domains alongside Cooking

To address the limited scope of our existing results, we expanded our current tests to include data from two other domains, simulating actors following LEGO instructions and wet lab protocols, respectively.

We discuss the integration of additional domains in Lines 67-75 in Section 1 (Introduction):

In order to test LLMs on emotionally contextualized goal-directed (ECGD) reasoning, RATF generates transcripts of simulated actors performing stepwise tasks across three domains (cooking recipes, LEGO builds, and wet lab protocols) and occasionally performing steps that evoke emotional reactions. For ease of discussion, we use standardized vocabulary across all domains: We refer to a set of step-by-step instructions as a recipe and the constituent inputs (i.e., ingredients, reagents, LEGO blocks) as ingredients. Initial recipe data comes from ground truth datasets per domain: RecipeNLG for cooking (Bie´n et al., 2020), Bricks for the Blind transcripts for LEGO (bri), and WLP-Dataset for wet lab protocols (WLP; Kulkarni et al., 2018).

Dataset processing of these domains is discussed on Lines 206-211 of Section 3.1 (Framework Design):

Each domain dataset is structured slightly differently, but we decided to but we decided to follow the formatting of RecipeNLG dataset, which contains a set of recipes, each with a title, a list of ingredients, and a list of natural-language instructions. We used gpt-6-astra\texttt{gpt-6-astra} to extract the ingredients and step-by-step instructions from the LEGO and WLP domains and reformat the dataset to resemble RecipeNLG; we refer to these datasets as ingredient-instruction datasets. The system prompts for this process can be found in Figure 13 of the Appendix.

Results for mistaken-ingredient and next action comparisons can be found in Figures 6 and 7 respectively. Below is a table of accuracies when performing mistaken-ingredient experiment averaged across models and broken down by domain, as shown in Figure 6:

SettingCookingLEGOWet Lab Protocol
2-option unambiguous0.0740.0950.072
4-option unambiguous0.3500.3860.136
2-option ambiguous0.0050.0560.060
4-option ambiguous0.0040.0560.016

We found that our results were consistent across domains, demonstrating that the behavior shown in our cooking results applies in broader instruction-following contexts. We hope these results narrow the gap between our evidence and conclusions.

W2: Related Work

We agree that some prior studies move beyond simple emotion classification, and we have updated our Related Work section to explicitly contrast RATF with DYNTOM and EmoBench based on key differences. We propose the following changes:

(Lines 133-140)

[While DYNTOM and similar ToM benchmarks] incorporate emotion, they add complexity along dimensions unrelated to our research question. In DYNTOM, models must track several mental states that are updated multiple times within a given scenario, including changes in emotion. This tests the model's ability to track states over long narratives, not whether models can leverage information about emotion to reason about goals. RATF isolates a single inference. The actor's mental state changes only once, at the end of the story when they make a mistake. The model must use the actor's emotional reaction to that mistake to determine which goal they were pursuing. The only source of difficulty in RATF is emotionally contextualized goal-directed (ECGD) reasoning itself, which tests the model's ability to apply emotional reasoning as a substep of overarching goal-directed reasoning.

(Lines 147-152)

However, Emobench only covers scenarios related to personal and social relationships, of the form "If [agent] feels [emotion], what should they do?", where the emotion is explicitly stated. We use RATF to generate step-by-step instruction following scenarios in three domains: cooking, LEGO building, and wet lab protocols. The framework can also be easily extended to domains outside of these three, giving RATF more coverage than Emobench and other benchmarks. Additionally, our framework requires LMs to identify the agent's emotion via the reaction and simultaneously determine the agent's goal based on their emotion.

We appreciate the feedback on this subject and would love to discuss this further.

AuthorsOpenReview ↗

Response from Authors (W3, Q1-2)

W3: Reworking of Visual Details

We have improved the visual clarity of Figures 6 and 7 (previously 5 and 6) as well as increased the size of Figure 3 to make the results more legible. We apologize for any confusion this may have caused.

Q1: Human Evaluation Data

We are currently in the process of running a human evaluation study where we test human performance on the 2- and 4-option settings of the mistaken ingredient test (both unambiguous and ambiguous variants) as well as the 10- and 20-in-context example settings of the next-action test. While humans are currently outperforming models, we are waiting on more data to provide conclusive results. We commit to include these results in our full report by the camera-ready deadline.

Q2: Next Action Disambiguation

As discussed in Line 351-353,

For each experiment, we adjust the number of occurrences of each mistake reaction in the full prompt to ensure the empirical distribution of mistake reactions is identical to the true mistake reaction distribution of the actor.

We modified the experiments in Section 3.2.2 (Personality-Based Next Step Prediction Methods) so "the models are asked to rank a list of three possible reactions from most to least likely" (Lines 343-344). The model can only reasonably be expected to learn the empirical frequencies from context as we do not provide any other information. Thus, the model's ranking of mistake reactions is evaluated on how well it matches the empirical distribution from the examples given in-context. We apologize for any confusion caused by unclear writing; these concerns have since been addressed in the revised paper in Sections 3.2.2 and 4.2 (Personality-Based Next Step Prediction Results).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.