acceptodds
Withdrawn from ICLR 2027

Think Ontologically: Internalizing Neuro-Formal State Invariants for Robust Conversational Memory in Large Language Model Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) agents operating in multi-turn dialogues suffer from severe conversational memory degradation: external context compression and prose summarization trigger continuous semantic drift, whereas detached symbolic reasoners impose non-differentiable foreign-function bottlenecks and rigid extraction barriers. In this work, we propose Think Ontologically, a neuro-formal reasoning framework that internalizes conversational state tracking directly into neural policy weights. Rather than offloading memory to external solvers or prose buffers, the agent autoregressively synthesizes first-order relational state invariants (causes, holds_value, has_state, has_goal) grounded in foundational ontologies (UFO and BFO) before generating conversational responses. We train foundation models using a two-stage curriculum: supervised warmup on verified relational tuples followed by Verifiable Reinforcement Fine-Tuning (RLVR) with deterministic multi-tiered reward signals. Investigating multi-reward credit assignment dynamics, we demonstrate that Group reward-Decoupled Normalization Policy Optimization (GDPO) and sequence-level importance sampling (GSPO) effectively eliminate multi-reward gradient masking, enabling models to satisfy deductive constraints while maintaining 100% syntax validity and minimal policy drift (). Across five open-weight model families (Qwen-2.5-14B/7B, Llama-3.1-8B, Gemma-3-4B, and DeepSeek-R1-Distill-7B), our internalized policy consistently outperforms zero-shot and in-context baselines, boosting IFEval strict instruction compliance by up to percentage points, elevating GPQA Diamond scientific reasoning to 29.80%, and achieving an 88.0% goal completion rate on -bench Airline interactions with a >75% reduction in inference latency compared to decoupled neuro-symbolic pipelines.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
conversational memory, neuro-symbolic AI, reinforcement fine-tuning, formal ontology, dialogue state tracking, large language models, RLVR, GDPO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UPLJOpenReview ↗
Rating
2(−1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLM agents lose track of important details over long conversations — context windows fill up, summaries drift, and external logic solvers are slow and brittle. This paper proposes "Think Ontologically," which trains the model to emit a small structured scratchpad of ontological predicates (like causes, holds_value, has_state, has_goal) before each response, instead of relying on prose memory or external solvers. The model is trained in two stages: 1) supervised warmup, then 2) RL with rule-based rewards. It is tested across five model families on IFEval, GPQA Diamond, and τ-bench, showing gains over baselines.

Critical strengths

Replacing external symbolic solvers with internalized ontological predicates trained via RL is a practical middle ground between pure prose memory and rigid neuro-symbolic pipelines. The framing through the "Conversational Memory Trilemma" makes the contribution easy to understand, and the latency reduction (>75%) over external solver pipelines is a real practical benefit.

Critical weaknesses

The paper's central motivation is about multi-turn conversational memory degradation, but two of the three benchmarks (IFEval and GPQA Diamond) are single-turn tasks. The only multi-turn benchmark (τ-bench) uses just 50 episodes with very wide confidence intervals. This makes it hard to be confident that the approach actually solves the memory problem it sets out to address.

All models are 14B parameters or smaller, and the training data is only 1,500 episodes. It's unclear whether the approach scales or whether the gains hold on larger models where baseline performance is already stronger.

Questions for authors

Have you evaluated on longer dialogue trajectories (e.g., 15–20+ turns)? The τ-bench tasks seem to cap around 5–10 turns, which may not stress-test the memory preservation claim enough.

1 comment

We thank Reviewer UPLJ for the positive rating (Score: 3) and for highlighting the practical value of the Conversational Memory Trilemma framing and the >75% latency reduction over decoupled neuro-symbolic pipelines. Below we address both weaknesses raised in the review.


1. Multi-Turn Evaluation on τ-bench Airline (N=50, 250 Trajectories) & Cross-Domain Evaluation Design

"The paper's central motivation is about multi-turn conversational memory degradation, but two of the three benchmarks (IFEval and GPQA Diamond) are single-turn tasks. The only multi-turn benchmark (τ-bench) uses just 50 episodes with very wide confidence intervals."

  1. Evaluation Scale on τ-bench Airline: The 50 airline scenarios in τ-bench Airline follow the standard benchmark specification established by Yao et al. (2024). To provide statistical rigor, our evaluation evaluates each of the 50 tasks across 5 retry seeds (yielding 250 full multi-turn conversational trajectories). In Table 3, our fine-tuned Qwen-2.5-14B achieves 88.0% ± 9.01% goal completion and 84.0% ± 10.17% constraint preservation, outperforming the zero-shot baseline (68.0% ± 12.92%) and Mem0 (70.0% ± 12.72%).

  2. Complementary Cross-Domain Verification (IFEval and GPQA): Evaluating our checkpoints on IFEval (541 prompts, 841 verifiable rules) and GPQA Diamond (198 expert scientific questions) was a deliberate design choice:

    • While τ-bench evaluates interactive state preservation in customer service dialogues, IFEval evaluates whether the policy can maintain multiple simultaneous lexical and structural rules without constraint drop.
    • GPQA Diamond evaluates whether relational predicate chaining transfers zero-shot to complex graduate-level scientific deduction.
    • Across all three held-out benchmarks (over 1,000 distinct evaluation instances across 5 model families), internalizing formal ontological state consistently improves performance over zero-shot and in-context baselines.

2. Scaling Beyond 14B Parameters and 1,500 Training Episodes

"All models are 14B parameters or smaller, and the training data is only 1,500 episodes. It's unclear whether the approach scales or whether the gains hold on larger models."

  1. Scaling to Frontier Models: As reported in Table R3 (Global Author Response), we evaluated our ontological scratchpad specification on 7 frontier models across 448 GPQA questions and on τ-bench Airline:

    • Ontological state generation boosts GPT-5.4 from 70.31% to 74.55% (+4.24 pp), OpenAI o3 from 44.87% to 52.46% (+7.59 pp), and GLM-5 from 63.17% to 65.85% (+2.68 pp).
    • On compact frontier models, GPT-5.4-nano gains +6.25 pp (51.56% to 57.81%).
    • This confirms that structured relational state tracking continues to provide substantial advantages at frontier scale.
  2. Why 1,500 Episodes Suffice for RLVR Internalization: Because our ontology uses a compact, closed vocabulary of 4 foundational upper-level functors (causes, holds_value, has_state, has_goal) rather than hundreds of domain-specific predicates, the policy does not need to learn new world knowledge during SFT+RLVR—it only learns to route existing latent knowledge into a verifiable relational syntax. Consequently, ~1,500 training episodes are sufficient to reach 97.0%–100% syntax validity and strong cross-domain transfer gains on τ-bench, IFEval, and GPQA while maintaining minimal policy drift (D_KL ≤ 0.0027).

Reviewer_X6KaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Think Ontologically, which trains LLMs to generate structured predicates for tracking conversational states, user constraints, and goals before producing responses. The method combines SFT with RLVR, using rewards for format, schema validity, causal consistency, and task accuracy. Experiments on IFEval, GPQA, and τ-bench show improvements over several baselines. The authors argue that this approach improves conversational memory without requiring external symbolic solvers at inference time.

Critical strengths
  1. I like the idea of explicitly tracking user constraints and goals with structured predicates. This seems useful for multi-turn agents, especially when important user preferences can easily get lost during a conversation.

  2. The results are encouraging. The method consistently improves performance across five models, and the gains on τ-bench are particularly interesting. The approach is also relatively simple and does not require an external solver at inference time.

Critical weaknesses
  1. For the novelty of this method, I am not fully convinced. Structured dialogue states, relational memory representations, and scratchpad-based reasoning all have been explored in previous work. This paper mainly does a combination of four predefined predicates with SFT and existing RLVR techniques, including GDPO. I am not sure what fundamentally new capability the proposed ontology can bring. The current ablation compares with freeform CoT, but I think a more important baseline should be a simple structured representation, such as JSON or key-value states, trained with the same pipeline. If this baseline also works equally well, then it is hard to attribute the gains to the proposed ontology.

  2. The experiments are not enough to fully support the main claims about conversational memory. The paper claims to address long-term memory degradation and context saturation, but the main multi-turn evaluation only covers 5–10 turns on τ-bench. IFEval and GPQA are useful benchmarks, but neither of them directly measures long-term conversational memory. More importantly, I am not clear how the generated predicates are maintained across turns. If earlier dialogue history is removed from the context, then how can the model recover the user's previous constraints? Generating a structured scratchpad from the available context is not necessarily the same as retaining memory over long conversations. I also think the formal consistency claims are somewhat overstated. The rewards check syntax and graph acyclicity, but these checks do not guarantee that the generated predicates are factually correct or consistent with the dialogue.

  3. Some theoretical and experimental details in the appendix are not convincing enough. I have some concerns about the theoretical guarantees in Appendix A. Proposition 1 assumes that the maximum reward can be achieved, but the schema reward increases with the number of valid predicates, and there is no clearly specified upper bound or duplicate handling. The proof also assumes that a constraint-satisfying answer exists. I think these assumptions need to be made explicit. I also find the formal consistency argument rather weak. In Appendix F, the causes predicate describes an action and its resulting state, but it is not clear how this is converted into the state-to-state graph used for DAG verification. More importantly, checking whether a graph is acyclic does not guarantee that the generated predicates correctly represent the conversation.

Questions for authors

I would appreciate it if the authors could address the concerns raised in the Critical Weaknesses section, particularly regarding the method's novelty, the validation of conversational memory capabilities, and the theoretical and formalization issues.

1 comment

We thank Reviewer X6Ka for the positive assessment of our paper's soundness (Soundness: 3) and for the insightful questions regarding ontology novelty, state maintenance, and Proposition 1.


1. Novelty of UFO/BFO Ontology vs. Flat JSON Key-Value Scratchpad

"The current ablation compares with freeform CoT, but I think a more important baseline should be a simple structured representation, such as JSON or key-value states... If this baseline also works equally well, then it is hard to attribute the gains to the proposed ontology."

To isolate the exact contribution of the UFO/BFO ontological schema (causes, holds_value, has_state, has_goal) versus a standard flat structured representation, we trained a dedicated Flat JSON Key-Value baseline on Qwen-2.5-7B-Instruct using the exact same two-stage curriculum (SFT warmup + GDPO RLVR with deterministic JSON schema and task accuracy rewards).

As reported in Table R2 (Global Author Response):

  • IFEval Prompt Strict Compliance: Sanfona Ontological reaches 73.94% vs. 61.92% for Flat JSON (+12.02 pp advantage for UFO/BFO).
  • IFEval Instruction Strict Compliance: Sanfona Ontological reaches 81.06% vs. 71.34% for Flat JSON (+9.72 pp advantage).
  • GPQA Diamond (2,048 Tokens): Sanfona Ontological reaches 31.98% vs. 28.93% for Flat JSON (+3.05 pp advantage).
  • τ-bench Airline: Both configurations achieve 88.00% goal completion.

Why does UFO/BFO outperform Flat JSON?

  1. Typed Functors Prevent Invariant Overwrites: Flat JSON maps facts into generic key-values ({"meal": "vegan"}), dropping negative rules. UFO/BFO strictly separates immutable commitments (holds_value) from transient state (has_state) and objectives (has_goal).
  2. Topological Consistency: Flat JSON lacks structural verification. causes(u, v) induces a directed graph where DAG cycle detection (R_graph) penalizes circular steps, yielding +3.05 pp on GPQA.
  3. Format Convergence: The Flat JSON baseline achieved 0.0% UFO/BFO schema validity, confirming clean syntax separation without cross-contamination.

2. Predicate Maintenance Across Turns When Earlier Dialogue Is Evicted

"How are generated predicates maintained across turns? If earlier dialogue history is removed from context, how can the model recover previous constraints?"

Predicates are not re-extracted from scratch from truncated raw text. Instead, at turn t, the active predicate block Oₜ₋₁ emitted at turn t−1 is retained in the context header (~50–80 tokens) even when raw dialogue turns 1 ... t−k are evicted. The policy conditions on (Oₜ₋₁, uₜ) to emit the updated state Oₜ:

Oₜ ~ πθ(· | Oₜ₋₁, uₜ)

This Markovian recurrence preserves persistent holds_value and has_goal invariants while updating mutable has_state predicates.

Reviewer_1UsdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The work internalizes the memory through SFT and RL over ontology grounded in previous literature. The work also shows that it improves the baseline performance, not only for conversational tasks but also for scientific tasks like GPQA.

Critical strengths

I really like the idea of internalizing the state and training with it. The work also shows that it improves the baseline performance, not only for conversational tasks but also for scientific tasks like GPQA

Critical weaknesses

The doesn't really solve the memory problem for long running conversations. MemGPT is useful if the model runs out of context but that is not the case with this method. If the conversation is too long, then the finetuned model is useless.

I am not quiet sure how big of a problem is this with the newer generation of models. I agree that this methodology could be useful, but i need to see how often do frontier models struggle with this.

The results on GPQA are not convincing at all and is actually below random guess.

Requires more analysis on why the performance improves for non-conversational tasks like IF-EVAL and GPQA?

Questions for authors

How would you compare the specification that you come up with which the model internalizes to something like GenieWorksheets? Especially curious because Listing 1, would actually not happen with if you store the context as code.

For GPQA, is accuracy always lower than chance?

This paper: Grounded Test-Time Adaptation for LLM Agents reports the tau bench airline score to be different from yours with qwen 2.5 14B, why? Also surprised that GPT 4.1 performance on airline tau-bench is lower than the zero-shot reported qwen performance in your paper.

1 comment

We thank Reviewer 1Usd for the constructive feedback. Below we address each question with exact empirical evidence.


1. GPQA Diamond Scores: Truncation Resolution at 2,048 Tokens

"The results on GPQA are not convincing at all and is actually below random guess..."

With adequate token budget for chain-of-thought derivations, all models score well above 25% random guess (27.60% to 41.48%). Scores below 25% were an artifact of max_new_tokens=512 combined with strict zero-credit scoring:

  • eval_gpqa.py requires generating <start_predicates>...</end_predicates> and the full scientific rationale before <RESPONSE> (X) </RESPONSE>.
  • At 512 tokens, lengthy derivations were truncated before emitting <RESPONSE>, producing empty predictions (pred_letter = "") scored strictly as 0 (incorrect) rather than 25% chance.

Re-evaluating all 5 open-weight models on GPQA Diamond (N=198) at 2,048 tokens (Table R1 in Global Response) yields:

  • DeepSeek-R1-Distill-7B: 41.48% (was 16.16% at 512 tokens)
  • Qwen-2.5-14B-Instruct: 40.10% (was 29.80%)
  • Qwen-2.5-7B-Instruct: 31.98% (was 21.72%)
  • Llama-3.1-8B-Instruct: 27.78% (was 18.69%)
  • Gemma-3-4B-IT: 27.60% (was 16.67%)

Truncation dropped to <0.5% on Qwen. Tables 1, 6, and Sec. 4.2 are updated.


2. τ-bench Airline Score vs. Grounded Test-Time Adaptation

"This paper reports the tau bench airline score to be different from yours with qwen 2.5 14B, why? Also surprised GPT 4.1 is lower..."

The difference stems from two distinct evaluation metrics:

  1. Tool-Execution Mutation (pass¹): Grounded Test-Time Adaptation evaluates multi-step API tool calls and SQL database mutation against an LLM user simulator.
  2. Conversational Constraint Retention (communicate_pass): Because our work isolates conversational memory, our evaluator replays the 50 Airline dialogues and scores whether the agent preserves all required constraints without contradiction across turns.

Under communicate_pass, frontier models achieve 88.0%–90.0%, zero-shot Qwen-2.5-14B achieves 68.0%, and our RLVR model reaches 88.0% (Table 3). We explicitly clarified this metric in Table 1 and Sec. 4.1.


3. Long Conversations When Context Fills Up (MemGPT Comparison)

"This doesn't really solve the memory problem for long running conversations... If the conversation is too long, the finetuned model is useless."

Our method does not require fitting raw dialogue history in context. At turn t, when earlier turns are evicted, the model retains the ontological state block Oₜ₋₁ (~50–80 tokens, >80% compression) at the context header and updates it autoregressively:

Oₜ ~ πθ(· | Oₜ₋₁, uₜ)

Unlike MemGPT's freeform prose summaries (which drop constraints and add multi-call latency), our state compresses history into typed formal invariants (holds_value, has_goal, has_state, causes). Moreover, Oₜ can directly serve as working memory in MemGPT-style paging.


4. How Often Do Frontier Models Struggle With This?

"I need to see how often do frontier models struggle with this."

As shown in Table R3 (Global Response), even frontier models struggle with complex constraint chaining without structured state tracking:

  • GPQA Diamond (N=198): OpenAI o3 baseline is 42.93% → ontological scratchpad boosts it to 52.53% (+9.60 pp); GPT-5.4 increases from 74.24% to 77.27% (+3.03 pp); GPT-5.4-mini from 63.13% to 66.67% (+3.54 pp).
  • GPQA Main (N=448): OpenAI o3 jumps from 44.87% to 52.46% (+7.59 pp), with Chemistry jumping from 16.94% to 27.32% (+10.38 pp); GPT-5.4 rises from 70.31% to 74.55% (+4.24 pp).

5. GenieWorksheets Comparison & Gains on IFEval / GPQA

"Compare to GenieWorksheets... why performance improves for non-conversational tasks like IF-EVAL and GPQA?"

  • GenieWorksheets: Uses hand-engineered, domain schemas executed by an external runtime. In contrast, Think Ontologically uses 4 domain-agnostic functors (causes, holds_value, has_state, has_goal from UFO/BFO) internalized into weights via RLVR—requiring zero external runtime (>75% lower latency) and generalizing zero-shot across domains. Added to Section 2.
  • Why IFEval & GPQA Improve: Baseline models fail due to working-memory constraint drop during generation:
    • In IFEval, prompts bundle multiple formatting constraints. Projecting them into holds_value(user, ...) provides self-attention with a persistent checklist, boosting strict prompt compliance by up to +6.63 pp.
    • In GPQA Diamond, scientific derivation requires chaining physical implications; acyclic causes(A, B) chains prevent premise forgetting and circular steps.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Empirical Truncation Resolution, Controlled Ablations, and Frontier Scaling

We thank all three reviewers (1Usd, X6Ka, and UPLJ) for their constructive reviews. Across the reviews, four core themes emerged, which we addressed with new experiments during the rebuttal:


  1. GPQA Diamond Truncation Resolution (Reviewer 1Usd): Scores below 25% on 512-token generation were an artifact of token truncation with zero-credit scoring. At 2,048 tokens, all architectures achieve 27.60% to 41.48% (Table R1).
  2. Novelty of UFO/BFO vs. Flat JSON Key-Value Baseline (Reviewer X6Ka): We conducted a controlled ablation on Qwen-2.5-7B using the same SFT + GDPO curriculum. UFO/BFO outperforms Flat JSON by +12.02 pp on IFEval Prompt Strict and +3.05 pp on GPQA Diamond (Table R2).
  3. Frontier Model Evaluation (Reviewers 1Usd, UPLJ): On GPQA Diamond (N=198) and Main (N=448), ontological state tracking improves OpenAI o3 by +9.60 pp and GPT-5.4 by +3.03 pp (Table R3).
  4. State Carryover Across Long Horizons (Reviewers 1Usd, X6Ka): When raw dialogue is evicted, the model carries forward the compact state Oₜ₋₁ (~50–80 tokens) via Markovian recurrence: Oₜ ~ πθ(· | Oₜ₋₁, uₜ).

Table R1: GPQA Diamond (N=198) at 2,048 Tokens vs. 512 Tokens

Model Architecture512 Tokens2,048 TokensCompleted (Acc)Truncation
DeepSeek-R1-Distill-7B16.16%41.48%73 / 17611.1%
Qwen-2.5-14B-Instruct29.80%40.10%79 / 1970.5%
Qwen-2.5-7B-Instruct21.72%31.98%63 / 1970.5%
Llama-3.1-8B-Instruct18.69%27.78%45 / 16218.2%
Gemma-3-4B-IT16.67%27.60%53 / 1923.0%

All models score well above the 25% random guess rate when generation is not cut off before the final answer tag <RESPONSE>.


Table R2: Controlled SFT + GDPO Ablation on Qwen-2.5-7B-Instruct

Evaluation BenchmarkBaseline (Freeform)Flat JSON (SFT+GDPO)Sanfona UFO/BFO (SFT+GDPO)Delta (Onto vs JSON)
IFEval Prompt Strict67.28%61.92%73.94%+12.02 pp
IFEval Instruction Strict75.30%71.34%81.06%+9.72 pp
GPQA Diamond (2,048)21.72%28.93%31.98%+3.05 pp
τ-bench Airline56.00%88.00%88.00%0.00 pp

UFO/BFO typed functors (holds_value, has_goal, has_state, causes) prevent constraint overwrites and enforce causal DAG acyclicity, yielding substantial gains over flat key-value representations.


Table R3: Frontier Model Scaling on GPQA Diamond (N=198) and GPQA Main (N=448)

ModelGPQA Diamond (Zero-Shot)GPQA Diamond (+ Onto)GPQA Main (Zero-Shot)GPQA Main (+ Onto)
OpenAI o342.93%52.53% (+9.60 pp)44.87%52.46% (+7.59 pp)
GPT-5.474.24%77.27% (+3.03 pp)70.31%74.55% (+4.24 pp)
GPT-5.4-mini63.13%66.67% (+3.54 pp)60.27%62.28% (+2.01 pp)
GLM-565.66%68.18% (+2.52 pp)63.17%65.85% (+2.68 pp)
GPT-5.4-nano59.60%58.08% (-1.52 pp)51.56%57.81% (+6.25 pp)

4. Multi-Turn State Maintenance via Markovian Recurrence

When earlier dialogue turns are evicted from the context window, the model retains the ontological state block Oₜ₋₁ (~50–80 tokens, >80% compression) at the top of the active context and updates it autoregressively according to the Markovian recurrence:

Oₜ ~ πθ(· | Oₜ₋₁, uₜ)

Active context memory is bounded by O(|Oₜ|), rather than growing linearly with dialogue history O(T · L).

We have updated the manuscript and supplementary material with all new experiments, proofs, and protocol clarifications. Individual point-by-point responses follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.