acceptodds
Under review as a conference paper at ICLR 2027

TextMani: A Constraint-Based Framework for Evaluating and Repairing LLM Reasoning from Language to Robotic Action

Abstract

Translating large language model (LLM) reasoning into robotic actions requires an interface between semantics and execution. This interface should make intermediate decisions inspectable and editable.Moreover, task success rates alone provide limited insight into task progress and points of failure throughout this process. We introduce TextMani, a unified framework for language-to-action execution, evaluation, and repair that uses explicit constraints as a shared intermediate representation. Given natural language instructions and scene features, TextMani generates execution targets through subgoal decomposition, feature selection, constraint formulation, and mathematical solving. Planning and control modules then translate these targets into robot actions. By linking intermediate artifacts to execution evidence, the framework supports process-level analysis based on physical subgoal progress and enables targeted repair following failures. Guided by execution feedback, it selects stages or constraints for modification, preserves unaffected content, and updates dependent downstream artifacts before re-execution. We evaluate 15 models across 18 simulated tasks and 10 models across six real-world robotic tasks. On real robots, TextMani improves task success from 27.2% with direct numerical-target prediction to 36.3%. Feedback-driven repair is also effective: Task-specialist repair recovers 47.7% of failed tasks compared with 24.5% for self-repair. Blind diagnosis reveals that failure recognition is the primary bottleneck.These results show that explicit executable specifications provide a common interface not only for comparing heterogeneous foundation models, but also for exposing where execution fails and enabling targeted physical recovery.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Language Models, Failure Diagnosis and Repair, Language-to-Action, Robotic Manipulation.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gBr4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TextMani connects foundation-model task reasoning to robot execution through explicit constraints and a shared grounding, solving, and control backend. Experiments in simulation and on real robots compare multiple models, separately evaluating initial execution, blind failure diagnosis, and repair given a failed subgoal. The main contributions are an inspectable common interface and a cross-model study of execution and correction.

Critical strengths
  1. The common constraint specification and execution backend make intermediate outputs inspectable and comparable across models, extending the analysis beyond final success rates.
  2. The experiments span multiple models, simulation, and real robots, with execution, blind diagnosis, and known-subgoal repair evaluated separately. Differences between error recognition and subsequent subgoal matching provide concrete evidence about the diagnostic bottleneck.
Critical weaknesses
  1. Constraint-level attribution and selective repair are not evaluated at the claimed granularity. Equations (5)–(8) use stable addresses to identify constraints for revision while preserving the rest, and this is listed among the main contributions. However, diagnosis measures error detection and agreement with a reference subgoal, while robot repair regenerates the entire specification of an already identified failed subgoal. The recovery rates of 24.5% and 47.7% establish that subgoal specifications can be repaired, but do not test erroneous-constraint localization, selective editing, or preservation of unaffected constraints. The distinctive repair mechanism therefore remains a method description rather than an empirically established contribution.
  2. The shared execution interface is insufficiently defined. The paper represents features, constraints, and mathematical expressions as a tuple and maps it to robot targets through a shared backend. It does not define a complete specification grammar, the scope and validity rules of MathIR, or the compilation and numerical-solving procedure. Figure 2 gives a useful example but does not replace these definitions, and the appendix mainly adds evaluation and accounting details. These omissions prevent readers from checking the executable contract followed by model outputs, identifying how much behavior is supplied by the backend, and reconstructing the mapping from constraints to motion targets.
Questions for authors
  1. Please provide an actual constraint-level repair trace showing the identified constraint addresses, expressions before and after revision, and the constraints left unchanged. Which reported results evaluate repair satisfying Equation (8)? If all current statistics come from regenerating the failed subgoal specification, please identify which fine-grained repair functions have been implemented and validated?
  2. Could you give a complete example for the grasping task in Figure 2: the actual structured model output, its compilation and solution into numerical targets, and the handling of invalid expressions or conflicting constraints? Please also identify which geometric features and action templates are supplied by the system and which elements the model selects or generates.
Reviewer_WkiiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TextMani, a constraint-based framework for aligning language model reasoning with robot manipulation. It breaks tasks into physical subgoals, where each subgoal is represented by physical evidence, semantic constraints, and mathematical expressions. A shared solver and controller then turn these specifications into robot actions. The authors evaluate 15 model configurations on 18 simulation tasks and 10 models on six real robot tasks. They measure both subgoal progress and final task success. They also test failure diagnosis and separately evaluate repair given a known failed subgoal. The paper reports higher real robot task success than direct target prediction. It also shows that models selected for each task also achieve better recovery than self-repair.

Critical strengths
  1. The proposed shared interface supports a systematic comparison of foundation models. It covers a variety of model configurations in both simulation and on real robots. Using the same downstream pipeline reduces implementation differences.

  2. The repair study presented in the paper includes 155 real robot failures. Given the failed subgoal, Self recovers 24.5% and Task-specialist recovers 47.7%. These results show that the proposed interface can support physical recovery for using foundation models in robotics.

Critical weaknesses
  1. The novelty is unclear. Prior works already covers constraint monitoring [1,2], specification correction [3], partial-progress evaluation [4,5], and diagnosis with recovery [6]. These papers are missing from the discussion of the contribution. The authors should inlcude these literature and do a detailed comparison.

  2. The experiments do not directly test constraint-level attribution or selective repair. Section 3.3 tests detection and subgoal localization, while Section 3.4 supplies the failed subgoal and regenerates its specification. These results do not establish the finer-grained capability claimed in Section 2.2.

  3. Lack of comparison with existing frameworks. The execution baseline is Direct target prediction. ReKep, PRoC3S, or OmniManip [7] should be compared under matched conditions. The benefits of the semantic representation and diagnostic feedback are also not isolated.

  4. The paper is hard to read and digest. The Introduction lacks clear paragraph breaks. The figures are dense, and Figure 1 does not clearly show the novelty.

References:

[1] Zhou et al. Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection. CVPR, 2025.

[2] Guo et al. DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment. IROS, 2024.

[3] Chen et al. AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers. ICRA, 2024.

[4] Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS, Datasets and Benchmarks Track, 2024.

[5] Yang et al. EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents. ICML, 2025.

[6] Yan et al. Can VLMs Diagnose and Recover from VLA Manipulation Faults? ICML, 2026.

[7] Pan et al. OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints. CVPR, 2025.

Questions for authors
  1. What is gained by separating semantic and mathematical representations? Can direct constraint generation or retries without diagnostic feedback achieve similar results with the same model and similar budget?
Reviewer_Xk1sOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

TextMani presents a unified constraint-based framework for language-guided robotic manipulation. It standardizes intermediate representations, including task-conditioned physical evidence, physical subgoals, semantic constraints, executable mathematical representations, and instantiated robot targets, to provide heterogeneous foundation models with a common specification and execution interface. The framework also links these representations to execution evidence to support failure diagnosis, constraint-level attribution, and localized specification repair.

The paper evaluates TextMani along three dimensions: executable specification generation, blind failure diagnosis, and scoped specification repair. Experiments cover 15 frontier-model configurations in simulation and 10 models across six real-robot manipulation tasks. The reported results show that TextMani improves real-robot task success over direct numerical-target prediction (36.3% vs. 27.2%). For blind diagnosis, models achieve 66.8% overall accuracy and 49.2% error recall, with a 99.2% subgoal-matching rate conditioned on successful error detection. For repair, task-specialist repair achieves a 47.7% recovery rate, compared with 24.5% for self-repair, given a known failed subgoal. Simulation results further indicate improvements in subgoal progress and the probability of achieving task success with increased recovery budgets.

Critical strengths

Overall, the paper contributes a common constraint-based language-to-action interface, an execution-grounded protocol for diagnosis and repair, and a multi-model evaluation of execution, diagnosis, and recovery capabilities. While the results demonstrate the feasibility and potential utility of the overall framework, the independent contributions of the constraint-based representation and scoped-repair mechanism remain insufficiently isolated by the current comparisons.

Critical weaknesses

1. The experiments do not sufficiently isolate the contribution of the proposed constraint-based representation.

Although TextMani outperforms direct numerical-target prediction on the evaluated real-robot benchmark, this comparison is between two overall workflows rather than a controlled ablation of the central design choices. The observed improvement could result from several components, including structured intermediate outputs, constraint formulation, numerical grounding, kinematic verification, or their combination. The current evidence therefore establishes the practical value of the complete pipeline more clearly than the independent value of the shared constraint interface. Without stronger ablations or competitive structured-target baselines, the novelty and necessity of the central representation remain uncertain.

2. The advantage of scoped specification repair over simpler recovery strategies is not established, and diagnostic reliability remains limited.

The repair experiments compare self-repair with task-specialist repair, but both use the proposed subgoal-scoped revision procedure. Consequently, the results primarily demonstrate the value of repairer selection, rather than showing that scoped repair is superior to full-task replanning, retrying the original specification, or other recovery strategies. Moreover, blind error detection has a reported recall of only 49.2%, limiting the reliability of an automated diagnosis-and-repair loop. The reported 99.2% subgoal-localization rate is conditional on successful error detection and should not be interpreted as overall localization accuracy. These limitations leave the central claim that explicit specification tracing enables more effective failure recovery insufficiently validated.

Questions for authors

1. Substantive concern: Insufficient evidence isolating the contributions of the proposed representation and repair mechanism

The current comparisons evaluate TextMani against direct numerical-target prediction and compare two repairer-selection strategies within the same scoped-repair workflow. However, these comparisons do not isolate the contributions of the proposed shared constraint-based specification interface or the scoped-repair mechanism. In particular, the improvement over Direct does not establish the independent benefits of explicit constraint representation, structured intermediate outputs, numerical grounding, or kinematic verification. Similarly, comparing two repairer-selection strategies within the same scoped-repair workflow does not establish whether scoped repair itself is more effective than alternative recovery strategies, such as full-task replanning or retrying the original specification. What controlled ablations or comparative evaluations support the specific contributions of these design choices beyond the performance of the integrated pipeline? Without such evidence, it remains difficult to assess the extent to which the results validate the paper's central methodological contributions, rather than primarily demonstrating the feasibility of the overall system.

2. Clarification: Ground-truth definitions and aggregation of diagnostic metrics Section 3.3 describes the error-detection protocol and reports the overall diagnostic metrics, but the definitions and aggregation procedures are not sufficiently clear from the presented description. How is the subgoal used to evaluate localization defined? Does it correspond to the first failed subgoal, the subgoal associated with an annotated anomaly, or another process-level label? How are the 99.2% conditional subgoal-matching rate and the 48.8% joint detection-and-localization rate calculated and aggregated across models and cases? Please also clarify the definitions and denominators of the normal-case identification rate (84.4%) and normal-case false-positive rate (12.8%), which appear inconsistent under standard definitions if computed on the same predictions. Finally, how are unparseable responses handled in the reported metrics?

3. Clarification: Independence of task-specialist selection from repair evaluation.

How are task-specialist models selected based on prior task performance, and are the data used for model selection independent of the failure cases used to evaluate repair performance? If the selection and evaluation data overlap, how is potential information leakage controlled? This clarification is important for assessing whether the reported 47.7% recovery rate reflects generalizable task-specific competence or potential information leakage from evaluation cases.

4. Clarification: Model autonomy and the implementation of the shared specification interface

Section 2.1 states that each foundation model independently generates its own subgoal decomposition and that model-generated specifications conform to a common structured specification space. Could the authors clarify how this independence is implemented in practice? Specifically, which aspects of subgoal decomposition and semantic constraint generation are determined by each model, and which are prescribed by TextMani through predefined structures, constraint templates, or other mechanisms? In addition, how are model-generated semantic constraints converted into executable mathematical representations, and what validation or conversion procedures ensure consistent downstream execution across models? These details are important for assessing the degree of planning autonomy retained by each model and the specific methodological contribution of TextMani's shared specification interface.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.