acceptodds
Withdrawn from ICLR 2027

Better Behavioral Prediction, More Faithful Model Ablations? Evidence from Sequential Choice

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Using predictive models to explain cognition requires more than accurate behavioral predictions. Input ablations offer an appealing route: remove information from a model and interpret the resulting performance change as evidence of its importance for behavior. Yet this inference assumes that the model's dependence on information reflects the dependence of the process generating the behavior. We test it in two synthetic sequential bandit tasks with known generating policies, where past choices can remain informative when feedback is unavailable to a predictor. We compare GRUs and Transformers trained from scratch, a fine-tuned LLaMA model, and cognitive models across systematically varied reward contributions. Our analyses distinguish prediction after training without reward observations from the response of a fixed predictor to donor-reward replacement. Three findings emerge. First, in the restless task, neural models trained without rewards predict held-out choices better than four simple training-fitted behavioral baselines. Second, under matched donor replacement, accurate predictors can respond much less than the known generator. Third, at some reward weights, neural networks predict better than a pooled reinforcement-learning model but have less faithful changes in choice probabilities; the model ordering differs between the two tasks. These independent-test results separate information sufficient for prediction from response fidelity under a specified ablation in sequential choice. They motivate validating model-ablation responses independently of predictive performance before using them to infer how the observed behavior was generated.

Primary area:
applications to neuroscience & cognitive science
Keywords:
computational cognitive modeling, sequential decision-making, model interpretability, input ablations, mechanistic inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A8o2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests if a small change in a behavioral model's predictions after reward removal allows the inference that the behavior generating process relies little on rewards. Authors used two tasks with known generating policies (a 4-armed restless bandit task that combines reward learning and last choice kernel, and a spatial correlated bandit that combines reward generalization with a local choice tendency). The authors vary reward contribution across seven weights. The tested models include GRUs, transformers and fine tuned Llama 3.1 70B, Rescorla-Wagner model, per participant fitted generator and an oracle that uses ground truth generating parameters. Study separates retraining without rewards from donor replacements. Each model's change in choice probabilities is then compared with oracle's - this constitutes response fidelity. Networks trained without rewards still predict choices well. In donor replacement, accurate networks respond far less than the generator. Prediction and fidelity rankings can also dissociate. The authors conclude that ablation responses must be validated separately from predictive accuracy before drawing conclusions about behavior-generating processes.

Critical strengths

A clear and careful separation of two manipulations (retraining without an input and fixed model perturbation), and a well designed response fidelity measure are major strengths of this paper.

Critical weaknesses

I found the major weaknesses to be lack of novelty/surprising result - given that choices encode past rewards and are held fixed under replacement, the networks' low sensitivity is largely expected, making the paper's contribution more about quantification of this than discovery. Also the paper shows rankings can flip between tasks but doesn't provide an explanation for why that might happen, so there's no way to predict when an ablation might mislead.

Questions for authors
  1. The oracle's epsilon greedy policy assigns a high probability to its best arm, so donor rewards make it confidently wrong which inflates delta NLL. Do the results change under a softer policy that's more typical for fitted human data?

  2. Is there a reason why generators didn't use parameters estimated from real human choices?

  3. Would it be possible to add larger spatial samples, as the RW-Llama comparison relies on only 50 task participants per weight?

  4. Do you have a possible explanation for why Llama responds more faithfully than RW in the spatial task but less in the restless task (eg. different meaning of w, RW's misspecification, or something llm-specific)?

Reviewer_2eULOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This is an interesting paper that studies the identifiability of different inputs as contributors to accurate modelling of human data. They find that when there are multiple interacting variables that are relevant to predicting a human choice (e.g. the reward history, and the choice history which itself depends on the reward history) then behaviour can often be modelled accurately even under conditions where the reward information is ablated (by shuffling assignments over participants). This is especially true for more powerful models (e.g. transformer / GRU-based models)

What the authors have done seems very sensible. They use a range of different models, which might plausibly be used for cloning behaviour on stylised behavioural experiments.

This is a nice results but it seems like a bit of a methodological footnote. Where you are conditioning on highly correlated / reciprocally dependent variables, it makes sense that you model will have identifiability issues. I do think it is potentially useful to highlight this, but I am not sure it merits an entire paper. I wasn't sure what the more general result or implication was.

I found the way the problem was described a bit hard to follow. I didn't really grasp what the paper was about until I got to the methods.

Critical strengths

This is a rigorous and carefully executed paper The results are presented clearly and are easy to follow

Critical weaknesses

The paper feels a bit niche. Some readers might find it useful, but it seems like a very specific point about an approach that not many people are using, and which those people may have clocked in any case? I don't know, I really tried to get excited about this paper but I failed.

Questions for authors

What are the more general implications of this finding? does it scale beyond modelling psychology experiments?

Reviewer_mmawOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the use of predictive yet uninterpretable models in cognitive science, such as RNNs and LLMs fine-tuned on human behavior. The paper considers the utility of input ablations, manipulations of the input to the model, as a way to understand the strategy used by a model for prediction, and as a check for whether the strategy is robust or whether the model predicts based on exploiting low-level regularities in human behavior. To test this, they train cognitive and neural models on synthetic RL tasks where the ground truth generator is known, and contrast training on the full signal (choices and rewards) from training on just choices. Overall, the paper argues that input-ablation results cannot be used to make inferences about the generating process (i.e., human behavior) if a model is simply predictive of the data under standard settings. Rather, the reliability of such results must be established separately.

Critical strengths
  • The paper tackles an interesting topic, of importance to cognitive science: While recent years have seen the rise of predictive (yet uninterpretable) models in cognitive, it remains a problem how to use these tools to advance the science of human behavior. The paper asks how to utilize highly-predictive models of human behavior to make inferences about the process that generated the behavior, and evaluates one behavioral path to do so: input ablations.
Critical weaknesses

While the general topic of the paper is interesting and relevant, unfortunately, I do not think that

  • First, the authors do not establish that input ablation is used by proponents of predictive modeling to make inferences on human behavior. Instead, they simply propose input ablation as a method, then spend the paper refuting their own suggestion. Without establishing that a particular type of inference is common, proposing an inference method then refuting it does not seem to me like a significant contribution.
  • Second, the input ablation method proposed by the author is quite specific (removing reward information in the input), and they do not consider methods of manipulating inputs that have been quite successful in neuroAI, such as finding inputs that maximally activate certain units in a network, or attempt to propose a different input ablation method that can reveal insights about the mechanism used by a model.
  • Third, the paper as a whole is quite difficult to read and AI was likely heavily used in generating the text, which can be seen in common phrases of the form "this is X, not Y" ("this intervention is a reward replacement, not literal deletion") often used by Claude.

Overall, the very specific input ablation and synthetic cases studied by the authors are unlikely to be of general interest to the community, and I believe the interesting question of what changes to an input can reveal insights about mechanism deserves to be studied more rigorously.

Questions for authors

None.

Comments on the paper

3 comments

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

Dear AC,

The authors' response to my review isn't showing up, and I can't see the responses to other reviews either. Could you confirm whether authors submitted their rebuttals?

Best regards

No worries. I just send a reminder to request the response if the reviews are avaliable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.