acceptodds
Withdrawn from ICLR 2027

Meta-Orchestration of Symbolic and Textual Agents for Personalized Decision-Making

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Easy decisions deserve simple models, while difficult decisions may benefit from richer reasoning. Utility-based models offer efficient and interpretable predictions, whereas large language models (LLMs) incorporate contextual information at greater computational cost. We propose Orchestration of Symbolic and Textual Agents (OSTA), a hierarchical meta-reasoning framework that adaptively combines utility models and LLMs for personalized decision modeling. Its orchestrator comprises two complementary stages: (i) Soft Inference Path Preference (SIPP), which assigns soft preferences between utility-only and mixed utility–LLM inference path; and (ii) Path-Conditional Agent Reliability (PCAR), which estimates agents' performance reliability on each path. A value-of-computation-inspired rule combines these signals with a tunable cost penalty to jointly select the inference path and agent, enabling performance and efficiency modes. Across six real-world datasets spanning transportation, health, and socioeconomic decisions, OSTA's performance mode achieves the best or tied-best performance on five of them, while efficiency mode retains an comparable performance, with approximately 91% fewer estimated tokens and 94% lower estimated inference cost for selected outputs. Population-scale case studies further illustrate OSTA's potential for resource-efficient personalized decision modeling. The source code will be fully released upon paper acceptance.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Symbolic Learning, Multi-agent System

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_binEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

OSTA selects among utility models and LLM based agents using estimated reliability and a cost penalty. Experiments on six decision datasets show improved prediction accuracy. The reported efficiency results count the cost of selected outputs in an evaluation with cached candidate answers.

Critical strengths

The results show useful differences between the utility models and LLM agents. Selecting among them improves average prediction accuracy. The appendix explains the calibration procedure in detail, and the ablations help identify which reliability signals contribute to performance.

Critical weaknesses
  1. The cost savings are not demonstrated for live inference. Appendix A.5 says that peer LLM answers are already available, while the reported cost counts only the selected answer.
  2. The routing comparison gives OSTA more information. It uses current candidate answers, while the baselines make selections with less information.
  3. The statistical reporting is confusing. Figure 4 refers to 60K test samples, while Section 4.2 describes bootstrap samples. The actual test sizes and resampling procedure need clarification.
Questions for authors

What are the accuracy and cost results when every LLM call needed for selection is charged? Does the advantage remain when all methods receive the same information at selection time?

Reviewer_CjEgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OSTA, a framework for routing between symbolic utility models and LLM-based predictors for personalized decision prediction via two main components: SIPP, which estimates a preference between inference paths using utility-gradient-based complexity and predictive confidence, termed “familiarity”; PCAR, which estimates candidate reliability using similarity to past successful cases, historical accuracy at similar prediction margins, and agreement among LLM-based agents. The framework combines path preference, estimated reliability, and a cost penalty to select a predictor. Experiments cover six survey-derived prediction tasks.

Critical strengths
  1. The paper provides relatively comprehensive component- and stage-level ablations.
  2. The evaluation includes several existing routing approaches and reports consistent improvements within the tested setting.
Critical weaknesses
  1. The evaluation is too limited to establish the framework’s added value. The tasks involve three-class prediction from 15–25 structured features, with little evidence that they require complex reasoning. Prediction difficulty could instead reflect missing personal context. Strong tabular baselines and evaluation with a stronger model pool are needed to establish whether the proposed orchestration offers meaningful benefits.

  2. The method is highly heuristic, with limited design justification or transferable insight. The framework combines multiple signals and weighting rules without adequately explaining their necessity. For example, why predictive confidence represents “familiarity,” or why path preference should multiply estimated correctness. The ablations demonstrate contributions within the chosen architecture, but do not establish an advantage over a simpler learned selector using the same information.

Questions for authors

Questions:

  • How does OSTA compare with a tuned tabular predictor and a simpler correctness-and-cost selector using the same available information?
  • What evidence shows that the complexity and familiarity signals identify cases that benefit from additional reasoning?
  • Could the authors provide representative complete inference inputs, including the learned textual context, to clarify the actual contextual and reasoning demands?

Suggestions:

  • Section 3.2 leaves essential definitions of the complexity components and their aggregation to Appendix A.1. Moving these short core formulas into the main text would make the method easier to follow and its design choices easier to assess.
Reviewer_Mt54OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes OSTA, a framework for predicting human choices by selecting among utility-only and utility-plus-LLM agents. Its contributions are a soft inference-path preference based on utility sensitivity and confidence, a path-dependent reliability estimator, and a selection rule combining these signals with a cost penalty. Evaluation covers six survey-based classification tasks, comparisons with individual models and adapted routers, component and stage ablations, bootstrap analyses, and population-scale cost projections.

Critical strengths
  1. Selecting between structured utility models and contextual LLM reasoning is a useful approach to human-choice prediction. OSTA makes this selection explicit and incorporates agent reliability rather than relying solely on model confidence.

  2. Performance mode achieves best or tied-best results on five of six datasets. Comparisons against the constituent agents and the ablations provide evidence that selecting complementary predictors adds value beyond using a single model.

Critical weaknesses
  1. Collective agreement requires predictions from other mixed agents; Appendix A.5 explicitly states that the experiments select among cached outputs. Consequently, selecting a utility-only output does not establish that LLM computation was avoided. Charging only the selected output understates the cost of generating predictions on demand and weakens comparisons with routers that select before generation. The population-scale projections inherit this limitation. Since computational savings are a central contribution, this substantially changes the significance of the work.

  2. Table 2 compares SIPP against random or validation-best path selection, but does not directly compare against selecting among all eight candidates using estimated correctness minus cost, with identical information and tuning. The B+PCAR variant already achieves average F1 of 0.65 versus 0.66 for the full method. A stronger matched comparison is needed to determine whether the hierarchy provides a meaningful advantage beyond the reliability estimator and its richer inputs.

Questions for authors
  1. What concrete deployment setting makes the peer predictions available without additional acquisition cost? Are the efficiency and population-scale claims intended to apply only to that setting?
  2. What cost–accuracy trade-off remains when every LLM call required to compute the routing features is charged? Alternatively, is there an evaluated acquisition procedure that avoids generating unselected outputs?
  3. How does OSTA compare with directly maximizing estimated correctness minus cost over all eight candidates, using the same information and calibration budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.