acceptodds
Under review as a conference paper at ICLR 2027

Operant Conditioning for Language-Model Agents

Abstract

Can language-model agents be conditioned by the consequences of their own actions? We introduce Operant Contingency Learning (OCL), which constructs policy targets from resource and burden changes while preserving their identity, direction, context, and current relevance. Across Qwen3-0.6B and Gemma-3-1B-IT with three seeds each, a controlled conditioning assessment recovers all four canonical operant effects. After 960 trials, positive and negative reinforcement increase held-out action probability by 21.50 and 17.75 percentage points relative to exposure-matched yoked controls, while positive and negative punishment decrease it by 10.16 and 9.10 points. All 24 model–seed–effect combinations change in the predicted direction relative to their calibrated initial policies, while yoked changes show no consistent direction. Separate skill-acquisition experiments use nine seeds per model on TextWorld and a simplified Telecom task derived from -bench. Under stochastic action selection, OCL-C without scalar loss achieves 77.15% error-free success on Telecom versus 59.65% for Scalar RL, and 14.55% on TextWorld versus 5.60% for Scalar RL. Both advantages occur in all 18 paired model–seed runs on each task and survive Holm correction across eight clean-success comparisons. OCL-C combined with scalar loss achieves 65.25% stochastic clean success on Telecom and 18.87% on TextWorld, revealing different observed rankings of the two OCL variants across tasks. In exploratory Telecom comparisons, OCL-C without scalar loss also reduces mean remaining burden by 45.2% relative to Scalar RL and reaches confirmed greedy mastery in 14/18 runs, compared with 11/18 for OCL-C combined with scalar loss and 5/18 for Scalar RL. During skill-acquisition, OCL-C without scalar loss exhibited mean action-probability changes in the predicted direction across all 16 task–model–consequence categories, compared with 14 for OCL-C with scalar loss and 13 for Scalar RL. These findings support typed action consequences as an effective basis for operant conditioning and task acquisition, including policy learning without a scalar task-reward loss.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, language-model agents, operant conditioning, post-training, structured feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_g2zJOpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
1
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This submission studies whether LLM agents can learn to increase/decrease beneficial/malign behavior via action. The authors proposes operant conditioned learning and classify the results from the action into four parts.

During experiments on Qwen3-0.8B, Gemma3-1B-1T, OLc-C are better than traditional scaler RL. The core insight would be: using the operantal changes as structural feedback would be an effective way to train LLM agents.

Critical strengths

The paper is closely related to psychological ideas/practices, and might provide insights for the RL practice when designing reward systems. Yet it is hard for me to grasp the idea behind the submission.

Critical weaknesses
  1. (still confusing/not consistent across the paper structure) Overall the paper's idea and organization is hard to absorb. The introduction is not clear during introducing the task, and the motivation. Also Figure 1 is abstract for me to comprehend: what does resource/burden means in the example?
  2. For the term operant, what is the differences between conditional generation? For auto-regressive transformer-based LLM decoding, single-sequence/multi-round interaction/long-horizon agentic task are all based on conditional generation. Previously generated tokens, actions will inference the activation status and the next token probability.
  3. (optimized) For the experiment, I think the baseline is not clear. For 'operant conditioning', what is its strength to current LLM+harness + standard decoding strategy? I failed to understand the experiment settings.
Questions for authors
  1. The author mainly experimented with skill-aquisition, which is abruptly switched to in line 60. Could the authors explain why skill-aquisition task is picked, but not other math reasoning/coding/agentic tasks? (Generalizability concerns, and I cannot catch the link between OCL and these specific task)

  2. What is the critical challenge in improving LLM's performance in skill-aquisition?

  3. Why persuing a none-scaler task-reward loss? From loss to gradient, methods like GAE would help estimate the advantages and

    I am open to discussions and would adjust my assessments.

7 comments
AuthorsOpenReview ↗

Briefly Expanding on Operant Conditioning

We would like to thank the reviewer for their thoughtful review and engaging with the material.

We provide below some examples and an intuition for understanding operant conditioning, which we hope will clear up any questions about the methodology before we share additional information about the method. 
Operant conditioning is often used in dog training. Here is an example and explanation for each of the 4 quadrants of operant conditioning (Figure 1):

Positive punishment: You are training your dog not to jump on people. You have your dog sit in front of you. You firmly place your foot on the leash. You wave your hands in its face, clap, make noise, and otherwise get your dog excited. Your dog jumps up and is quickly corrected by the leash, causing them to be forced back down.

  • This is “positive” because we are applying a stimulus (the force of the leash on the dog’s neck is added to the context/state/situation). This is a “punishment” because the dog stops jumping on people when they are excited.

Positive reinforcement: In the same situation as above, eventually and in most cases, the dog will stop jumping up while you are trying to make them excited. Once this happens, you are able to give your dog a treat. Giving your dog a treat for behavior you want to see is positive reinforcement.

  • This is “positive” because the treat is a stimulus added to the situation. This is “reinforcement” because the dog eventually has all four paws on the floor when they are greeting people.

Negative punishment: Let’s imagine we are playing with our dog at the park. They begin barking at other dogs or exhibiting otherwise unwanted behavior. We remove our dog from the park (put them in “time-out”) to communicate that their behavior is inappropriate.

  • This is “negative” because we are removing the dog from the situation. The dog loses access to the park, which we assume (in this imaginary situation) is something that they want. This is also "punishment" because eventually the dog stops misbehaving at the park.

Negative reinforcement: Negative reinforcement is often the trickiest to explain, but imagine we are teaching our dog to sit and that they already know, more or less, how to behave when we tell them to sit. We will lift up on the leash and add burden using pressure-based training and then give the dog the “sit” command. As soon as the dog’s butt hits the floor, we will release the pressure on the leash. A similar example is a car’s seatbelt chime stopping when you put your seatbelt on.

  • This is “negative” because we are removing a burden that already exists (the uncomfortable pressure of the leash). This is “reinforcement” because the dog becomes more likely to sit when formally commanded.

Important note: None of these would be examples of punishment or reinforcement without decreases or increases in the behavior that the intervention is designed to modify. E.g. - if the dog is no less likely to misbehave when removed from the park, that is not negative punishment. If we are no more likely to put on our seatbelt when the seatbelt chime stops, then that is not negative reinforcement. Punishment and reinforcement effects are exactly that: decreases and increases in exhibited behavior.

In all of these situations, scalar-reward reinforcement learning can combine different consequences into a single number, obscuring their separate identities. Operant conditioning (and our method "Operant Contingency Learning") retains the typed consequence feedback in a mathematical form that allows for an agent to experience this nuance during the learning process.

Conditional generation: All 3 methods we tested in this manuscript use conditional generation and choose actions based on context. “Operant” refers to the learning process, or what happens after actions are made. In OCL, the consequence of an action changes how likely the learner will choose that action again in the same situation in the future. This is not necessarily always going to be the case for traditional reinforcement learning.

We hope this illustration will be insightful and we are happy to answer any additional questions relating to general understanding of the topic. We will add these examples to the final manuscript so that readers may understand operant conditioning more easily. We will also follow up later in the discussion period to answer your remaining questions and provide additional results.

AuthorsOpenReview ↗

**Explaining Skill Acquisition, Baselines, and the Agentic Task Datasets**

This comment addresses parts of:

  • W1 (what do resource and burden mean and how are they relevant to each task?)
  • W3 (why is operant conditioning interesting or helpful for solving these tasks?)
  • Q1 (what is skill acquisition?), and
  • Q2 (what makes this challenging?)

Q1. Skill acquisition.

Here we mean "learning to complete an interactive task reliably through a sequence of actions." Both tasks (TextWorld and Telecom) are agentic tasks. Learners are given a text-based representation of the current state, choose an action, observe the change in the environment, and then proceed. We chose these tasks because their state transitions allow us to measure what each action adds, removes, damages, makes unavailable, etc. This creates a controlled test of consequence-based learning before testing on coding, mathematics, or more abstract assessments.

Q2. Why these tasks are challenging, and how they test operant behavior.

The critical challenge here is “credit assignment.” Task rewards (Scalar RL) are delivered at the end of an episode, so the learner must figure out which of its actions produced which consequences. An action can help in one context but hurt in another.

Some benchmarks allow for task completion regardless of unintended consequences or collateral damage. Agents might be allowed to complete a task in a way that causes harm, wastes an opportunity, or loops over actions that no longer have benefit. OCL uses intermediate consequences to modify the probability that a certain action will happen. We then measure task success ("skill acquisition") and whether consequences are able to move action probabilities in the predicted directions.

For example, when assessing behavior, does a punished action become less likely? Does a rewarded action become more likely? Our intuition might tell us "yes, of course," but our ablations show that the only methods to exhibit such behavior on average in all tested situations were "OCL-C no scalar" and its "no scalar proportional" ablation (newly uploaded manuscript, charts available on page 36).

W3. What exactly is the baseline? For each model—seed pair, Scalar RL and OCL share the task interface, action-selection procedures, and all start from the same calibrated policy. This “LLM + harness + standard decoding” protocol is used for each method. The only differences are what each method receives (consequences, rewards, punishment, burden, etc) and how each method uses that information (the math for policy construction and update form). Comparing Scalar RL (optimizing the policy to achieve task success) to OCL (additionally optimizing the intermediate behavior) helps to explain the difference in performance for how each method turns its experience into policy updates. We will share in later comments results from our new baselines that match OCL's information, in addition to ablations for OCL.

W1. Below, we explain each dataset's task.

Task 1: Telecom.

Models must learn to resolve a service problem in this modified τ²-bench task. Our simplified version gives the model a service problem and public state. Available actions include:

  • inspecting the state,
  • requesting a setting change,
  • and finishing.

For example, the agent may choose an acton such as enabling or disabling a feature like "mobile data." Correct actions will lead to successful device configuration and, when possible, remove service faults. Wrong answers or state changes are violations. Success (or "skill acquisition") requires completing the task without violations within 15 actions. Verified configuration (intermediate success) grants the model resource. Service faults and violations cause the model to accrue burdens. As an example: repairing a service fault removes a burden, which might cause the learner to exhibit negative reinforcement.

Task 2: TextWorld.

Models learn to operate a relay station. Our TextWorld-inspired environment requires passing a POWER gate and a SAFETY gate. The POWER gate requires having "charge," while clearing the SAFETY gate will cause "damage" if active "contamination" remains in the environment. The agent sees:

  • charge,
  • contamination,
  • credits,
  • tool prices,
  • gate order,
  • and previous action feedback.

Its set of available actions include:

  • four colored tools,
  • advance,
  • and wait.

Tools add or remove charge or contamination, and their effects reverse between POWER and SAFETY contexts (i.e. - a tool might add contamination during a POWER gate, but remove contamination during a SAFETY gate). Charge is a resource; contamination is a burden. The agent must learn which tool produces which consequence, whether that consequence is currently useful, and when to stop working. Success requires completing both gates without violations and within 15 actions.

We agree that our submitted manuscript could have more clearly explained these tasks. We will add these explanations to the final paper.

AuthorsOpenReview ↗

Adding Detail to Motivations

We thank the reviewer again and briefly connect the 1st comment to the reviewer's Question 3 (why a non-scalar loss? why not use GAE to estimate advantage?).

Q3a. Motivation for a Non-Scalar Loss:

  1. In any situation where something (a tool, a stimulus, some sort of intervention) is added or removed, scalar-reward reinforcement learning can combine different consequences into a single number, obscuring their separate identities. Operant conditioning (and our method "Operant Contingency Learning") retains the typed consequence feedback in a mathematical form that allows for an agent to experience this nuance during the learning process.
  2. Adding the scalar task-reward loss to OCL lowered performance on Telecom (77.15% vs. 65.25% stochastic) and on TextWorld at 512 groups (62.8%/36.6% vs. 23.7%/21.8% greedy/stochastic).
  3. In our TextWorld ablation, the consequence operator was shown to be OCL's most important component. Without consequence information and only applying scalar rewards to the individual actions, the "No consequences" ablation scored 8.0% greedy and 6.0% stochastic success on the TextWorld agentic task at 512 groups averaged across 9 seeds and 2 models. With the consequence operator, OCL scored 62.8% greedy and 36.6% stochastic success. These results are significant, with a Holm adjusted p value of 0.008.

Q3b. Why Not GAE?:

  1. GAE uses a learned critic to estimate each action’s advantage from the scalar reward. It does not supply information about the consequences of an agent's actions in a given context or state from a behavioral psychology standpoint (addition or removal of resource, burden, tools, obstacles, etc).
  2. We did not run a baseline with a learned critic. Our closest baseline is "Own-decision Vector RL," which receives the scalar task reward as well as every consequence, each credited directly to its corresponding action. "OCL-C no scalar" outperforms "Own-decision Vector RL" on TextWorld at 512 groups (+43.7 / +16.4 points greedy/stochastic, Holm significant). Removing the consequences from OCL while keeping the reward information ("No consequences") performs like Scalar RL.

Additional Baselines:

Entering into the end of the discussion period, we remain available to address any additional concerns you might have, or elaborate on any concerns that remain unresolved.

Reviewer_EyyyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work try to introduce Operant Contingency Learning (OCL), a language model agent training method inspired by operant conditioning. Instead of relying solely on scalar task rewards, OCL leverages structured consequences arising from actions to guide policy learning. Overall, this paper demonstrates the feasibility of constructing policy training objectives from structured action outcomes, as well as its performance gains on specific tasks. However, the experiments are primarily limited to small-scale models and simplified environments. Additionally, the work lacks ablations of key components and baselines with matched supervision information, making it impossible to clearly disentangle performance improvements attributable to the algorithmic mechanism itself from those brought by additional consequence supervision.

Critical strengths

S1. Clear research motivation: It investigates whether structured action consequences can outperform conventional scalar rewards to guide language agent learning, and the research question is well aligned with agent training.

S2. Well-specified methodology: The OCL framework integrates consequence types, contextual relevance, and action memory into policy objectives, with detailed mathematical descriptions of training objectives and update procedures.

S3. Rigorous controlled experimental design: It adopts exposure-matched yoked controls to disentangle action–consequence contingency effects from mere consequence exposure, improving the reliability of experimental conclusions.

Critical weaknesses

W1: The Resource herein is not autonomously identified and defined by the Agent but pre-specified by researchers. Whether variables are treated as resources or burdens, and under what circumstances resource gains are valuable, are governed by environment design and preset relevance conditions. Consequently, OCL mainly learns to update policies based on pre-defined fluctuations of resources and burdens, rather than independently identifying task-relevant values. One limitation of this method is that it focuses on policy learning from structured outcomes, instead of enabling agents to learn reward functions or discover task values autonomously.

W2: (Section 4, p.5, lines 266–269;Section 7, p.9, lines 462–465) In addition to terminal rewards, OCL leverages fine-grained action consequences, relevance signals, and opportunity-loss estimates, whereas scalar RL relies primarily on terminal scalar rewards. As such, the performance gap between the two approaches arises from differences in both training methodology and the amount of supervisory information available. This implies that it remains unclear whether the performance gains stem from the objective formulation of OCL or the extra supervisory signals it incorporates. To resolve this ambiguity, I recommend adding a scalar RL baseline equipped with access to identical consequence information, for instance via reward shaping based on intermediate rewards.

W3: (Section 3, pp.3–5;Section 7, p.9, lines 462–465) Ablation experiments lack evaluations on individual key components. OCL-C jointly utilizes contextual memory, relevance modeling, target construction, opportunity feedback and terminal credit assignment, yet the individual contribution of each component to the final success rate has not been validated. This implies that although the full method achieves superior results, it remains unclear which components are essential, or whether simple adjustments to target probabilities could yield comparable performance.

W4: The experimental tasks are relatively simplified, limiting the generalizability of the conclusions. The experiments only involve two small-scale language models and short-horizon simulation tasks with constrained action spaces. TextWorld is an adapted task, while Telecom is a local simulation constructed from materials related to τ²-bench, rather than a fully standardized benchmark. This implies that while the current results demonstrate the effectiveness of OCL within these controlled environments, it remains unknown whether the findings can generalize to larger models, open action spaces, or complex real-world agent tasks. We recommend validating the method in environments closer to standard benchmarks and explicitly delimit the scope of the paper’s conclusions.

Questions for authors

W1–W3 most directly impact the attribution of the method’s strengths and innovative mechanisms in this work.

4 comments
AuthorsOpenReview ↗

**New baselines, ablations, and a longer TextWorld budget**

We would like to thank the reviewer for their thoughtful review and engaging with our material.

This comment addresses W2 and part of W3. Charts for new methods and their behavior can be found in the updated paper on the last page (page 36).

Our Limitations section named two gaps: ablations, and baselines with the same information as OCL. We have now run both, and we are posting the results early so there is time for discourse. We will respond to your remaining points in a separate comment later in the discussion period.

Five new methods (baselines, ablations)

  • Vector RL: Scalar RL plus the same consequences (including opportunity loss and relevance/affordability gating) on its own trajectories. The discounted return credits each consequence to that action plus every earlier action.
  • Own-decision Vector RL: Vector RL crediting each consequence only to its corresponding action.
  • "No memory" ablation: "OCL-C no scalar" without the memory counterpart (v=0v = 0 in Eq. 4). Tests consequence memory.
  • "No consequences, No memory" ablation: "OCL-C no scalar" without consequences, memory, and outcome targets on every decision.
  • Weight-Space Editing (WSE): our adaptation of Takehi et al. (2026), "Sparks of Weight-Space Programming" (https://rikiyat.github.io/blog/weight-space-programming/). An LLM editor (gpt-6-astra) edits the policy's weights directly. It gets the same consequences, episode budget (768 Telecom, 2,048 TextWorld) and evaluation as OCL. WSE uses an external editor and had shorter runtimes.

256 → 512 TextWorld budget. The 256-group budget was set by compute constraints prior to submission. Appendix B.4 (p. 27) suggested "OCL-C no scalar" was under-trained: Gemma seed 13 rose from 7% to 70% dev success between groups 192 and 256 and ended at 65.6%, against 13.5–37.5% for the other Gemma seeds. We therefore extended all methods to 512 groups on the same data. All 18 "OCL-C no scalar" runs improved from 256 to 512, greedy and stochastic.

Results. Clean success, “OCL-C no scalar” minus each method (greedy/stochastic). Means over 9 seeds and 2 models; tests use 9 paired seed blocks, averaging model differences within each seed. Bold: Holm-adjusted p < .05.

<“OCL-C no scalar” differences, % points><Telecom, 192><TextWorld, 256><TextWorld, 512 (post hoc)>
"No consequences, No memory"+8.9 / +22.6+15.8 / +8.9+54.7 / +30.7
"No memory"−0.2 / +3.6−1.1 / +0.3+6.4 / +4.5
Scalar RL+8.8 / +17.5+11.9 / +9.0+51.0 / +30.4
Own-decision Vector RL+4.5 / +10.8+1.9 / −6.4+43.7 / +16.4
Vector RL+26.0 / +3.4−3.8 / −7.5+35.7 / +11.1
WSE+38.2 / +6.7+5.4 / −4.9+44.6 / +16.9

Families: Scalar RL and Vector RL each use eight tests at original budgets (2 tasks * 2 modes * 2 OCL variants) and four post hoc tests at TextWorld 512. Each other comparator uses two tests per task–budget pair; WSE families are post hoc.

Findings.

  1. Consequence-based targets matter. "No consequences" ablation retains the fitter but uses outcome targets on every decision. It performs worse than "OCL-C no scalar" in every reported comparison.
  2. How typed consequences are delivered matters. Both Vector RL variants outperform OCL-C no scalar on TextWorld 256 stochastic success; at 512 groups, “OCL-C no scalar” reaches 62.8%/36.6% greedy and stochastic, against 27.1%/25.5% for "Vector RL" and 19.1%/20.3% for "Own-decision Vector RL." These gains are Holm significant.
  3. Goal-directed scalar loss can interfere with learning. OCL-C with scalar loss stays around 23.7%/21.8% greedy and stochastic at 512 groups.
  4. Memory redistribution helps some. (TextWorld 512 stochastic, +4.5). Most of the advantage over Scalar RL remains without counterpart redistribution ("No memory" ablation). Memory redistribution did not add much for Gemma, while Qwen’s development mean of seeds at 512 is 33% for “no-scalar proportional” vs. 55% for original “OCL-C no scalar,” suggesting the benefits of memory-selected action probability redistribution may differ across models.
  5. Operant behavior during training. Only “OCL-C no scalar” and its "No memory" ablation show mean action-probability changes in the predicted direction in all 16 [task * model * consequence] categories. Neither Vector RL variant shows the predicted direction after burden removal in TextWorld. WSE runs did not collect information to judge its operant capabilities.
  6. Information-matched Weight-Space Editing Plateaus. WSE’s TextWorld greedy success barely moves after doubling groups from 256 to 512 (17.1% compared to 18.2%), while “OCL-C no scalar” improves from 22.5% to 62.8%.

We will add these results to the final paper. We will address additional weaknesses and concerns raised by your review later in the discussion period.

AuthorsOpenReview ↗

Addressing Questions and Weaknesses Individually

This comment partially addresses:

  • W1 (“Resource” is pre-specified by researchers)
  • W2 (Information-matched baseline)
  • W3 (Missing ablations)
  • W4 (Non-standard benchmarks)

W1: “Resource” is pre-specified by researchers: We agree this is a limitation, and it is also a design choice. The method is inspired by dog training, where actions often experience immediate consequences administered by the trainer. We hypothesize that explicitly mapping actions and states to consequences could support more responsible and predictable agent behavior. Our results support reduced measured harm in these tasks: OCL-C no scalar achieved lower mean remaining burden on Telecom compared to Scalar RL and lower mean damage on TextWorld at 512 groups than Scalar RL and both versions of Vector RL. Learning resource and burden definitions and contextual relevance rules would extend OCL beyond its present researcher-specified criteria. We agree that this would be a valuable contribution to the literature. We will add this as a limitation of our design in the manuscript.

W2: Information-matched baseline: We agree that our original “Scalar RL” was insufficient a baseline to adequately separate OCL’s target construction from the information it received. Here, we refer to our prior comment including our newer Vector RL and Own-decision Vector RL baselines, in addition to our Weight-Space Editing baseline. Each of these methods received the same consequence feedback as OCL, yet were unable to break out of the plateau at 512 groups on TextWorld (additionally see page 36 in the updated manuscript for development results, Figure 9 — WSE results are not in the chart and instead contained in the previous comment above). Please let us know if this does not address your information-matched baseline concerns.

W3: Missing ablations: We provide “Outcome-only OCL” to cover your “Target construction” ablation point. It tests whether or not fitting outcome-based targets on every decision is sufficient, which was worse than OCL in every reported comparison. We have also run the contextual memory ablation. Its results were provided in our 1st comment (it is “no scalar proportional”). We are currently running “no terminal credit,” “no opportunity loss,” and “no relevance” on TextWorld at 512 groups. We will share results once the runs have completed (likely later this week).

W4: Benchmarks are nonstandard: At the time of running these experiments we had not identified a ready-to-use language-agent benchmark exposing all four action-linked resource and burden changes together with the contextual relevance measurements required for operant conditioning. We identified this as a gap in the literature, though this also required us to modify existing datasets and environments to make this paper possible. While this is indeed a limitation, we believe this also idenitfies a unique opportunity for researchers interested in dataset design. However, we understand we might have missed a suitable benchmark during our literature review. We will note this as a limitation.

Regarding model size: please also note that ILQL (ICLR 2023), GLAM (ICML 2023), DPO (NeurIPS 2023), and 1-shot RLVR (NeurIPS 2025) all included experiments with models 1.5B parameters or fewer. Smaller models helps to facilitate achieving meaningful confidence intervals with multiple seeds. However, we will note this as a limitation of our experimental protocol in the manuscript. If time and budget allow, we will attempt to run a slightly larger model on a single seed, though this may not be possible within the period of the discussion.

If any of these answers are unsatisfactory, please let us know as soon as possible and we will do our best to address any remaining concerns.

AuthorsOpenReview ↗

Ablation Results

Below, we provide 9 seed ablation results for TextWorld at 512 groups for both greedy and stochastic evaluation on both the original models Qwen 0.6B and Gemma 1B. Positive is better for "OCL-C no scalar," Negative is better for the ablation. Holm significant results are bolded (same comparison policy noted in the original appendix and 1st discussion comment). "No consequences" and "No memory" ablations were originally provided a week ago on November 6th, but we include them again here for completeness.

Greedy

<“OCL-C no scalar” differences, % points><TextWorld, 192><TextWorld, 256><TextWorld, 512>
No consequences, No memory+6.9+15.8+54.7
No memory−2.4−1.1+6.4
No terminal credit−2.0+0.8+9.5
No opportunity loss−0.4+1.4−2.1
No relevance−0.7+1.2−3.4

Stochastic

<“OCL-C no scalar” differences, % points><TextWorld, 192><TextWorld, 256><TextWorld, 512>
No consequences, No memory+7.8+8.9+30.7
No memory+0.3+0.3+4.5
No terminal credit−1.6−1.6−1.3
No opportunity loss+0.4+0.5+0.2
No relevance+0.4−1.5−8.8

Findings:

  1. The consequence operator is the most important component: removing consequences from OCL and relying on scalar credit applied to intermediate actions yielded large, significant results, with meaningful effects at all 3 evaluation budgets (192, 256, and 512). Both greedy and stochastic performance for "OCL-C no scalar" outperform "No consequences" by more than double the ablation's performance at 512 groups.
  2. Relevance hurt the stochastic policy: At 192 and 256 groups, relevance had no significant effect, but 512 shows an 8.8 point improvement in the stochastic policy without relevance annotations. Impact on greedy evaluation was not significant.
  3. Memory helped the final stochastic policy: OCL-C consequence memory improved the stochastic policy significantly at 512 groups, with a relative ~14% improvement (36.6% compared to 32.2%).
  4. Terminal credit results were noisy: The greedy +9.5 looks like a large effect, but the Holm p value indicated insignificance at 0.391. 5 wins to 4 losses for "OCL-C no scalar" vs. "No terminal credit" on greedy policy evaluation. Removing terminal credit did help a little for Stochastic at 192 groups.

With less than a week left in the discussion period, we remain available to address any remaining concerns you might have.

AuthorsOpenReview ↗

Manuscript Updates, Additional Evidence

We would like to thank the reviewers and we have uploaded a revised manuscript. We have made the updates to reflect the changes you have asked to see in the breadth of comparison and depth of mechanistic analysis. The changes also aim to address information density concerns and general writing complexity to make the work more accessible to its intended audience.

  1. The main text has been rewritten to cover the following points:
    1. Abstract density: The abstract is no longer as dense (reviewer 6Pqz W5, reviewer g2zJ W1)
    2. Operant Conditioning Examples: We include examples for all four quadrants of operant behavior (p2) (reviewer 6Pqz W5, reviewer g2zJ W1)
    3. Method Naming: Methods are now named after their closest architectural comparators, and we remove the scalar loss note, replacing it with +PG for policy gradient
      1. Scalar RL → REINFORCE
      2. Vector RL → Shaped Reward REINFORCE
      3. Own-decision Vector RL → Own-Decision Shaped Reward REINFORCE
      4. OCL-C → OCL+PG
      5. OCL-C no scalar → OCL
    4. Nomenclature: We rename “skill acquisition” to “agentic task” to resolve g2zJ’s concern
    5. Mechanistic Comparisons: We add a table showing mechanistic relationship to prior methods (Table 1 p.4) for ease of reader understanding, noting reviewer 6Pqz’s W5 (dense writing) and reviewer g2zJ’s concerns about presentation and informativeness
    6. Updated Figures: We add new figures to explain:
      1. Example TextWorld and Sepsis episodes (Figures 3-4, pp6-7)
      2. Updated training diagram noting naming changes (Figure 2, p5)
      3. How each ablation builds its targets (Figure 14, p40)
    7. New Task, Larger Model: "Sepsis" is our new exploratory experiment that aims to address concerns relating to model size and task complexity (pp41-42, Figures 16-17, Table 12)
      1. Our Sepsis simulator experiment from Oberst and Sontag 2019 [1] ran SR-REINFORCE vs. OCL on a single seed using model Qwen3-1.7b. OCL took more than 1.4M actions and SR-REINFORCE roughly 0.5M actions. OCL scored 61% normalized return compared to the information-matched baseline’s policy collapse at -5.4% return. 0% return is random guessing. OCL improved +10 points from group 8 to group 384. Training included nearly 100,000 episodes.
    8. Ablations: We have added 6 ablations in total - some on both TextWorld and Telecom, others only on TextWorld. All are 9 seeds and both models (Table 10, p40, Tables 3-4 for Telecom p29 and p31):
      1. No memory, No types
      2. No memory, No consequences
      3. No memory
      4. No relevance
      5. No opportunity loss
      6. No terminal credit
    9. Harm comparisons: we aggregate harm (damage, violations, burden) at final budgets and compare across the 5 primary methods (Table 11, p41).

Should any concerns remain, we kindly request your clarification so that we may address them before the discussion period ends.

[1] Oberst, M. & Sontag, D.. (2019). Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:4881-4890 Available from https://proceedings.mlr.press/v97/oberst19a.html.

Reviewer_6PqzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Operant Contingency Learning (OCL), which uses changes in resources and burdens to construct policy targets for LLM agents. The authors test the four operant effects using yoked controls, then compare OCL with a REINFORCE-style baseline on simplified TextWorld and Telecom tasks.

I like the idea, particularly the attempt to test whether behavior changes because of the action-consequence relation. The statistical reporting is also fairly careful. My main concern is that OCL receives substantially more feedback than the baseline. The results look promising, but I am not sure how much they tell us about the proposed target construction. I also found the method difficult to follow in places.

Critical strengths

Interesting perspective connecting behavior analysis with LLM agent training.

The use of exposure-matched yoked controls is a nice design and rarely seen in this area.

I appreciate that the paper reports the higher damage on TextWorld. This complicates the story, but it is useful to see where the method does worse.

Critical weaknesses

The conditioning experiment seems to follow largely by construction, since the policy is directly fit toward targets that raise or lower the chosen action. Running a scalar-reward learner in the same yoked assay would show whether the effects are specific to OCL.

OCL gets much richer feedback than the reinforce baseline, including an opportunity-loss signal that seems to rely on information the agent can’t observe. The comparison is not fair without a baseline using the same information.

The “without scalar loss” claim is somewhat misleading, as a large part of the training targets on Telecom still come from terminal reward.

Experiments are limited to two small models with six fixed actions , which does not support claims about general LLM agents.

The writing is very dense. Key terms are defined only in the appendix, and the abstract has too many numbers.

Questions for authors
  1. Have you compared with a scalar RL baseline that uses a shaped reward built from the same consequence signals?
  2. How much do the acquisition results change when opportunity-loss feedback is removed? Separately, what happens if the no-scalar variant receives no terminal credit?
  3. How sensitive are the results to the hyperparameters, such as the 0.4 target scale and the memory rate?
5 comments
AuthorsOpenReview ↗

**New baselines, ablations, and a longer TextWorld budget**

We would like to thank the reviewer for their thoughtful review and engaging with our material.

This comment addresses W2 (baseline using the same information), and Q1. Charts for new methods and their behavior can be found in the updated paper on the last page (page 36).

Our Limitations section named two gaps: ablations, and baselines with the same information as OCL. We have now run both, and we are posting the results early so there is time for discourse. We will respond to your remaining points in a separate comment later in the discussion period.

Five new methods (baselines, ablations)

  • Vector RL: Scalar RL plus the same consequences (including opportunity loss and relevance/affordability gating) on its own trajectories. The discounted return credits each consequence to that action plus every earlier action.
  • Own-decision Vector RL: Vector RL crediting each consequence only to its corresponding action.
  • "No memory" ablation: "OCL-C no scalar" without the memory counterpart (v=0v = 0 in Eq. 4). Tests consequence memory.
  • "No consequences, No memory" ablation: "OCL-C no scalar" without consequences, memory, and outcome targets on every decision.
  • Weight-Space Editing (WSE): adaptation of Takehi et al. (2026), "Sparks of Weight-Space Programming" (https://rikiyat.github.io/blog/weight-space-programming/). LLM editor (gpt-6-astra) edits policy's weights directly. It gets the same consequences, episode budget (768 Telecom, 2,048 TextWorld) and evaluation as OCL. Editor compute is not matched.

256 → 512 TextWorld budget. The 256-group budget was set by compute constraints prior to submission. Appendix B.4 (p. 27) suggested "OCL-C no scalar" was under-trained: Gemma seed 13 rose from 7% to 70% dev success between groups 192 and 256 and ended at 65.6%, against 13.5–37.5% for the other Gemma seeds. We therefore extended all methods to 512 groups on the same data. All 18 "OCL-C no scalar" runs improved from 256 to 512, greedy and stochastic.

Results. Clean success, “OCL-C no scalar” minus each method (greedy/stochastic). Means over 9 seeds and 2 models; tests use 9 paired seed blocks, averaging model differences within each seed. Bold: Holm-adjusted p < .05.

<“OCL-C no scalar” differences, % points><Telecom, 192><TextWorld, 256><TextWorld, 512 (post hoc)>
"No consequences, No memory"+8.9 / +22.6+15.8 / +8.9+54.7 / +30.7
"No memory"−0.2 / +3.6−1.1 / +0.3+6.4 / +4.5
Scalar RL+8.8 / +17.5+11.9 / +9.0+51.0 / +30.4
Own-decision Vector RL+4.5 / +10.8+1.9 / −6.4+43.7 / +16.4
Vector RL+26.0 / +3.4−3.8 / −7.5+35.7 / +11.1
WSE+38.2 / +6.7+5.4 / −4.9+44.6 / +16.9

Families: Scalar RL and Vector RL each use eight tests at original budgets (2 tasks * 2 modes * 2 OCL variants) and four post hoc tests at TextWorld 512. Each other comparator uses two tests per task–budget pair; WSE families are post hoc.

Findings.

  1. Consequence-based targets matter. "No consequences" ablation retains the fitter but uses outcome targets on every decision. It performs worse than "OCL-C no scalar" in every reported comparison.
  2. How typed consequences are delivered matters. Both Vector RL variants outperform OCL-C no scalar on TextWorld 256 stochastic success; at 512 groups, “OCL-C no scalar” reaches 62.8%/36.6% greedy and stochastic, against 27.1%/25.5% for "Vector RL" and 19.1%/20.3% for "Own-decision Vector RL." These gains are Holm significant.
  3. Goal-directed scalar loss can interfere with learning. OCL-C with scalar loss stays around 23.7%/21.8% greedy and stochastic at 512 groups.
  4. Memory redistribution helps some. (TextWorld 512 stochastic, +4.5). Most of the advantage over Scalar RL remains without counterpart redistribution ("No memory" ablation). Memory redistribution did not add much for Gemma, while Qwen’s development mean of seeds at 512 is 33% for “no-scalar proportional” vs. 55% for original “OCL-C no scalar,” suggesting the benefits of memory-selected action probability redistribution may differ across models.
  5. Operant behavior during training. Only “OCL-C no scalar” and its "No memory" ablation show mean action-probability changes in the predicted direction in all 16 [task * model * consequence] categories. Neither Vector RL variant shows the predicted direction after burden removal in TextWorld. WSE runs did not collect information to judge its operant capabilities.
  6. Information-matched Weight-Space Editing Plateaus. WSE’s TextWorld greedy success barely moves after doubling groups from 256 to 512 (17.1% compared to 18.2%), while “OCL-C no scalar” improves from 22.5% to 62.8%.

We will add these results to the final paper. We will address additional weaknesses and concerns raised by your review later in the discussion period.

AuthorsOpenReview ↗

Addressing Questions and Weaknesses Individually

This comment partially addresses:

  • W1 (conditioning experiment follows by construction)
  • W2 (opportunity loss ablation)
  • W3 (misleading claims re: “without scalar loss”)
  • W4 (small models, scope)
  • W5 (writing density)
  • Q1 (information-matched baseline)
  • Q2 (skill acquisition without opportunity-loss feedback, no terminal credit)
  • Q3 (HPO tuning, sensitivity)

W1. Yoked Assay Results Follow By Construction
 We expect that Scalar RL would likely exhibit similar behavior, though we have not tested this. We do not aim to claim OCL is the only method capable of exhibiting operant behavior. We believe the results observed during training on Telecom and TextWorld are more indicative of each training method’s ability to elicit operant behavior in complex situations, and offer the chart on page 36 as complementary evidence of mean response directions during task learning. Scalar RL still achieved only 13 of 16 even at 512 groups of TextWorld, and “OCL-C no scalar” and its “no scalar proportional” ablation were the only two methods to exhibit 16 of 16 [task—model—consequence] categories in the expected direction.

W2. Richer Feedback, Opportunity Loss Ablation

We assert that “opportunity loss” is environment feedback similar to a game letting a player know they can no longer complete an objective. The agent learns that an action cost it future progress. It is the task’s form of negative punishment (lost access). Both Vector RL baselines receive annotations computed by the same rule on their own trajectories, addressing the information difference in the original comparison (Q1). We have started full scale 9 seed 512 group TextWorld ablations for “OCL-C no scalar” with your requested “no opportunity loss,” “no terminal credit,” and reviewer Eyyy’s requested “no relevance” ablation. We have chosen TextWorld because it is less saturated than Telecom. If time and budget permit, we will run “no terminal credit” on Telecom (to your point W3).

W3. Misleading Claims

To elaborate, the mechanism was designed so that the two loss terms (scalar and typed) would be additive and could be easily ablated by setting either to 0. We agree that “without scalar loss” could send the wrong message. We will rename “OCL-C no scalar” to “OCL-C (no PG),” defining PG early in the text as the separate task-reward policy-gradient term.

W4. Small Models, Scope

We agree that our model sizes are a scoping limitation and concede this issue will not be able to be resolved during the discussion period. We would like to offer a counterpoint for your consideration: new RL algorithms are often published using smaller models, as this makes it more affordable to achieve meaningful confidence intervals on multi-seed runs. For example, ILQL (ICLR 2023), GLAM (ICML 2023), DPO (NeurIPS 2023), and 1-shot RLVR (NeurIPS 2025) included experiments with models of 1.5B parameters and below. However, we agree that our work could have benefitted from using larger models. We will note in the final manuscript that our conclusions are limited to these model sizes and constrained action spaces.

W5. Writing Density

We agree that the paper could be better organized for its intended audience. Our submission assumed more familiarity with operant conditioning than it should have. We will reorganize the paper to introduce key terms in the main text, introduce the tasks before the mechanism, and reduce numerical density in the abstract.

Q1. Information Matched Baseline

Here we reference the previous comment that provided the information-matched Vector RL, Own-decision Vector RL, and other baselines and ablations.

Q2. Ablations

As noted in our answer to W2, we are running ablations to address this question re: opportunity loss and terminal credit. However, we would like to point to the table in our previous comment: “Vector RL” and “Own-decision Vector RL” receive the same opportunity-loss information and terminal task reward, yet were still far behind on TextWorld at 512 groups. This shows that access to that information does not reproduce the result. The 9-seed ablations we are running (no relevance, no term credit, no opportunity loss) will show their contribution within OCL.

Q3. HPO Tuning & HP Sensitivity

All methods used fixed hyperparameters without a tuning search. We agree that searching around 0.4 for target scale and around 0.5 for memory rate would help practitioners better understand tradeoffs in their choices. We will attempt a small tuning run on target scale and memory rate, though we may not be able to run it on 9 seeds given compute limitations. We will identify that as a limitation.

If any of these answers are unsatisfactory, please let us know as soon as possible and we will do our best to address any remaining concerns. We will provide additional ablations on TextWorld once the results complete, though these results will likely not be ready until the end of this week.

AuthorsOpenReview ↗

Ablation Results

Below, we provide 9 seed ablation results for TextWorld at 512 groups for both greedy and stochastic evaluation on both the original models Qwen 0.6B and Gemma 1B. Positive is better for "OCL-C no scalar," Negative is better for the ablation. Holm significant results are bolded (same comparison policy noted in the original appendix and 1st discussion comment). "No consequences" and "No memory" ablations were originally provided a week ago on November 6th, but we include them again here for completeness.

Greedy

<“OCL-C no scalar” differences, % points><TextWorld, 192><TextWorld, 256><TextWorld, 512>
No consequences, No memory+6.9+15.8+54.7
No memory−2.4−1.1+6.4
No terminal credit−2.0+0.8+9.5
No opportunity loss−0.4+1.4−2.1
No relevance−0.7+1.2−3.4

Stochastic

<“OCL-C no scalar” differences, % points><TextWorld, 192><TextWorld, 256><TextWorld, 512>
No consequences, No memory+7.8+8.9+30.7
No memory+0.3+0.3+4.5
No terminal credit−1.6−1.6−1.3
No opportunity loss+0.4+0.5+0.2
No relevance+0.4−1.5−8.8

Findings:

  1. The consequence operator is the most important component: removing consequences from OCL and relying on scalar credit applied to intermediate actions yielded large, significant results, with meaningful effects at all 3 evaluation budgets (192, 256, and 512). Both greedy and stochastic performance for "OCL-C no scalar" outperform "No consequences" by more than double the ablation's performance at 512 groups.
  2. Relevance hurt the stochastic policy: At 192 and 256 groups, relevance had no significant effect, but 512 shows an 8.8 point improvement in the stochastic policy without relevance annotations. Impact on greedy evaluation was not significant.
  3. Memory helped the final stochastic policy: OCL-C consequence memory improved the stochastic policy significantly at 512 groups, with a relative ~14% improvement (36.6% compared to 32.2%).
  4. Terminal credit results were noisy: The greedy +9.5 looks like a large effect, but the Holm p value indicated insignificance at 0.391. 5 wins to 4 losses for "OCL-C no scalar" vs. "No terminal credit" on greedy policy evaluation. Removing terminal credit did help a little for Stochastic at 192 groups.

We believe this only leaves hyperparameter tuning unanswered. With less than a week left in the discussion period, we remain available to address any remaining concerns you might have.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.